Evaluating Job Exposure to Large Language Models
7GPTs are GPTs
Speaker: Daniel Rock
Summary
========
Daniel discusses the findings of their paper, Generative Pre-trained Transformers are General Purpose Technologies, and the broader implications of AI systems in the workforce. They address the question of whether AI systems will negatively impact employment and explore the idea that large language models, such as Generative Pre-trained Transformers (GPT), can significantly transform the economy. The speaker takes an optimistic stance, stating that they do not believe there will be massive job displacement due to these technologies. They observe a shift towards non-routine cognitive tasks and introduce a new approach to evaluating the exposure of tasks to large language models.
Using a new rubric-based approach on tasks from the O*NET database, we quantify the labor market impact potential of LLMs. Both human annotators and GPT-4 assess tasks based on their alignment with LLM capabilities and the capabilities of complementary software that may be built on top of these models with our rubric. Our findings reveal that between 61 and 86 percent of workers (for LLMs alone versus LLMs fully integrated with additional software) have at least 10 percent of their tasks exposed to LLMs. Additional software systems have the potential to increase the percentage of the U.S. workforce that has at least 10 percent of their work tasks exposed to the capabilities of LLMs by nearly 25 percent. We find that LLM impact potential is pervasive, LLMs improve over time, and complementary investments will be necessary to unlock their full potential. This suggests LLMs are general-purpose technologies. As such, LLMs could have considerable economic, societal, and policy implications, and their overall impacts are likely to be significantly amplified by complementary software.
Topics
=====
⃝ Impact of AI systems on employment
* The speaker believes that there will not be massive job displacement due to AI systems.
* There is a shift towards non-routine cognitive tasks.
* A new approach to evaluating the exposure of tasks to large language models is introduced.
⃝ Evaluation of exposure to large language models
* The speaker and their team used the O*NET database to evaluate the exposure of tasks.
* Tasks were categorized into ‘No Exposure (E0)’, ‘Exposure with LLM (E1)’, and ‘Exposure with LLM + other software (E2)’.
* Human annotators provided “opinions” about the exposure level of various tasks within Jobs. The results were compared about GPT4 predictions and used as the baseline of the evaluation. There is agreement between humans and GPT-4 in evaluating tasks at the occupation level.
* Approximately 80% of workers may have around 10% of their tasks exposed to large language models.
⃝ Impact of automation on jobs
* Certain types of workers are more exposed to automation assuming no significant change in the subset of tasks associated with their job and not enough resources for reskiling / upskilling.
* Automation can potentially improve job satisfaction by removing mundane tasks and shifting cognitive energy to more creative and demanding ones.
* Job descriptions, list of subtasks in a job, are dynamic and creating the right infrastructure is important for safe and responsible adoption of technology.
* The current study focuses on task level assessment of exposure and future work could explore system level exposure by including more high level dependencies.
* It is difficult to advice on the necessary policy solutions but the importance of evaluation and quality control is clear.