Performance Assessment of Text Similarity Algorithms Through Classification Metrics for BSIT Graduate Job Concordance

Abstract

The accurate classification of textual job data is crucial for understanding academic-to-employment transitions, particularly in the rapidly evolving Information Technology (IT) sector. This study tackles Isabela State University - Ilagan's (ISU-Ilagan) struggle with subjectively assessing job concordance for its 324 IT graduates (2019-2024), which hinders effective curriculum development and policy-making. Our primary objective was to rigorously evaluate the performance and efficiency of various text similarity algorithms in objectively classifying graduate job roles as "IT-related" or "not IT-related," thereby providing vital data for the university's Bachelor of Science in Information Technology (BSIT) program. Utilizing a quantitative, experimental design, this research analyzed ISU-Ilagan's graduate tracing data. Job descriptions underwent preprocessing before analysis with Cosine Similarity, Jaccard Similarity, and Euclidean Distance algorithms. Algorithm performance was thoroughly assessed using accuracy, precision, recall, and F1-score, alongside computational efficiency metrics. Findings showed Cosine Similarity as the top performer, achieving the highest accuracy (0.935), exceptional precision (0.986), a strong F1-Score (0.952), and superior computational efficiency. Euclidean Distance also performed well (accuracy: 0.910, precision: 0.952, F1-Score: 0.932), sharing identical recall (0.863) with Cosine Similarity, though it was slightly less efficient. Jaccard Similarity yielded lower metrics and efficiency. Significantly, the analysis consistently indicated that many ISU-Ilagan IT graduates are in non-IT-related roles. This study provides crucial objective data for ISU-Ilagan. Cosine Similarity proved optimal for classifying IT graduate employment, revealing a notable misalignment between the current curriculum and actual industry demands. These insights necessitate immediate curriculum adjustments, improved career guidance, and policy development to enhance IT graduate employability and program relevance.