(LOS ANGELES) When a human says an event is “probable” or “likely,” people generally have a shared, if fuzzy, understanding of what that means. But when an AI chatbot like ChatGPT uses the same word, it’s not assessing the odds the way we do, my colleagues and I found.
We recently published a study in the journal NPJ Complexity that suggests that, while large language model AIs excel at conversation, they often fail to align with humans when communicating uncertainty. The research focused on words of estimative probability, which include terms like “maybe,” “probably” and “almost certain.”
kAmqJ 4@>A2C:?8 9@H px >@56=D 2?5 9F>2?D >2A E96D6 H@C5D E@ ?F>6C:42= A6C46?E286D[ H6 F?4@G6C65 D:8?:7:42?E 82AD 36EH66? 9F>2?D 2?5 =2C86 =2?8F286 >@56=D] (9:=6 E96 >@56=D 5@ E6?5 E@ 28C66 H:E9 9F>2?D @? 6IEC6>6D =:<6 “:>A@DD:3=6[” E96J 5:G6C86 D92CA=J @? 96586 H@C5D =:<6 “>2J36]” u@C 6I2>A=6[ 2 >@56= >:89E FD6 E96 H@C5 “=:<6=J” E@ C6AC6D6?E 2? g_T AC@323:=:EJ[ H9:=6 2 9F>2? C6256C 2DDF>6D :E >62?D 4=@D6C E@ edT]k^Am
kAm%9:D 4@F=5 36 3642FD6 9F>2?D 42? :?E6CAC6E H@C5D DF49 2D “=:<6=J” 2?5 “AC@323=6” 32D65 >@C6 @? 4@?E6IEF2= 4F6D 2?5 A6CD@?2= 6IA6C:6?46D] x? 4@?EC2DE[ =2C86 =2?8F286 >@56=D >2J 36 2G6C28:?8 @G6C 4@?7=:4E:?8 FD286D @7 E9@D6 H@C5D :? E96:C EC2:?:?8 52E2[ =625:?8 E@ 5:G6C86?46D H:E9 9F>2? :?E6CAC6E2E:@?D]k^Am
kAm~FC DEF5J 2=D@ 7@F?5 E92E =2C86 =2?8F286 >@56=D 2C6 D6?D:E:G6 E@ 86?56C65 =2?8F286 2?5 E96 DA64:7:4 =2?8F286 FD65 7@C AC@>AE:?8] (96? 2 AC@>AE 492?865 7C@> “96” E@ “D96[” E96 px’D AC@323:=:EJ 6DE:>2E6D @7E6? 3642>6 >@C6 C:8:5[ C67=64E:?8 k2 9C67lQ9EEADi^^5@:]@C8^`_]`_bg^Dc`dge\_ad\_hdg`\KQm3:2D6D 6>365565 :? :ED EC2:?:?8 52E2k^2m] (96? 2 AC@>AE 492?865 7C@> t?8=:D9 E@ r9:?6D6[ E96 px’D AC@323:=:EJ 6DE:>2E6D @7E6? D9:7E65[ A@DD:3=J 5F6 E@ 5:776C6?46D 36EH66? t?8=:D9 2?5 r9:?6D6 :? 9@H A6@A=6 6IAC6DD 2?5 F?56CDE2?5 F?46CE2:?EJ]k^Am
k7:8FC6mk:>8 2=ElQ2 >F=E:4@=@C E9C66\A2?6 8C2A9:4 H:E9 :4@?D C6AC6D6?E:?8 9F>2?D 2?5 C@3@ED[ 2?5 E6IE 2?5 2CC@HDQ DC4lQ9EEADi^^:>286D]E964@?G6CD2E:@?]4@>^7:=6D^fa__ff^@C:8:?2=^7:=6\a_ae_aac\df\F:ec<E];A8n:I=:3lC3\c]`]_U2>AjBlcdU2>Aj2FE@l7@C>2EU2>AjHlfc_U2>Aj7:El4=:AQ ^mk7:842AE:@?mpx 492E3@ED 5@?’E :?E6CAC6E ‘AC@323=J’ 2?5 ‘>2J36’ E96 D2>6 H2J J@F 5@] |2J2?< z6;C:H2=k^7:842AE:@?mk^7:8FC6m
k9am(9J :E >2EE6CDk^9am
kAmu2C 7C@> 36:?8 2 =:?8F:DE:4 BF:C<[ E9:D >:D2=:8?>6?E :D 2 7F?52>6?E2= 492==6?86 7@C k2 9C67lQ9EEADi^^5@:]@C8^`_]`__f^D``aah\_ab\_cbef\_Qmpx D276EJk^2m 2?5 9F>2?\px :?E6C24E:@?] pD =2C86 =2?8F286 >@56=D 2C6 :?4C62D:?8=J FD65 :? 9:89\DE2<6D 7:6=5D =:<6 k2 9C67lQ9EEADi^^5@:]@C8^`_]`_bg^Dc`dh`\_ad\_c_fc\JQm962=E9 42C6k^2m[ k2 9C67lQ9EEADi^^5@:]@C8^`_]`_bg^Dc`cef\_ad\e`bcd\dQm8@G6C?>6?E A@=:4Jk^2m 2?5 k2 9C67lQ9EEADi^^HHH]E:>6D9:896C65F42E:@?]4@>^@A:?:@?^H6\>FDE\D6E\CF=6D\2:\FD6\D4:6?E:7:4\HC:E:?8\2?5\A66C\C6G:6HQmD4:6?E:7:4 C6A@CE:?8k^2m[ E96 H2J E96J 4@>>F?:42E6 C:D< 364@>6D 2 >2EE6C @7 AF3=:4 ECFDE]k^Am
kAmx7 2? px 2DD:DE2?E 96=A:?8 2 5@4E@C[ 7@C :?DE2?46[ 56D4C:36D 2 D:56 67764E 2D “F?=:<6=J[” 3FE E96 >@56=’D :?E6C?2= 42=4F=2E:@? @7 “F?=:<6=J” :D >F49 9:896C E92? E96 5@4E@C’D :?E6CAC6E2E:@?[ E96 C6DF=E:?8 564:D:@? 4@F=5 36 7=2H65]k^Am
k9am(92E @E96C C6D62C49 :D 36:?8 5@?6k^9am
kAm$4:6?E:DED 92G6 DEF5:65 9@H 9F>2?D BF2?E:7J F?46CE2:?EJ k2 9C67lQ9EEADi^^A9:=2C49:G6]@C8^2C49:G6^zt}(~tQmD:?46 E96 `he_Dk^2m[ 2 7:6=5 A:@?66C65 3J rxp 2?2=JDED E@ :>AC@G6 :?E6==:86?46 C6A@CE:?8] |@C6 C646?E=J[ E96C6 92D 366? 2? 6IA=@D:@? :? =2C86 =2?8F286 >@56= =:E6C2EFC6 D66<:?8 E@ =@@< k2 9C67lQ9EEADi^^5@:]@C8^`_]`gedb^G`^a_ad]7:?5:?8D\24=]g`bQmF?56C E96 9@@5k^2m @7 ?6FC2= ?6EH@C<D E@ 36EE6C F?56CDE2?5 E96:C “3692G:@CD” 2?5 =:?8F:DE:4 A2EE6C?D]k^Am
kAm~FC DEF5J 255D 2 =2J6C @7 4@>A=6I:EJ 3J EC62E:?8 E96 :?E6C24E:@? 36EH66? 9F>2?D 2?5 2CE:7:4:2= :?E6==:86?46 2D 2 3:@=@8:42=\=:<6 DJDE6> H96C6 >62?:?8 42? 568C256] xE >@G6D 36J@?5 D:>A=J >62DFC:?8 :7 2? px :D “D>2CE” 2?5 :?DE625 2D<D :7 :E :D 2=:8?65]k^Am
kAm~E96C C6D62C496CD 2C6 4FCC6?E=J 6IA=@C:?8 H96E96C D@\42==65 k2 9C67lQ9EEADi^^DA64ECF>]:666]@C8^492:?\@7\E9@F89E\AC@>AE:?8Qm492:?\@7\E9@F89E AC@>AE:?8k^2m – 2D<:?8 E96 px E@ D9@H :ED H@C< – 42? 7:I E96D6 6CC@CD] w@H6G6C[ @FC DEF5J 7@F?5 E92E 6G6? 25G2?465 C62D@?:?8 5@6D?’E 2=H2JD 3C:586 E96 82A 36EH66? DE2E:DE:42= 52E2 2?5 G6C32= =236=D]k^Am
k9am(92E’D ?6IEk^9am
kAmp 8@2= 7@C 7FEFC6 px 56G6=@A>6?E :D E@ 4C62E6 >@56=D E92E 5@?’E ;FDE AC65:4E E96 ?6IE =:<6=J H@C5 3FE 24EF2==J F?56CDE2?5 E96 H6:89E @7 E96 F?46CE2:?EJ E96J 2C6 4@?G6J:?8] #6D62C496CD 2C6 42==:?8 7@C >@C6 C@3FDE 4@?D:DE6?4J >6EC:4D E@ 6?DFC6 E92E :7 2 >@56= D66D 2 `_T 492?46 :? E96 52E2[ :E 49@@D6D E96 D2>6 H@C5 6G6CJ E:>6]k^Am
kAmpD H6 >@G6 E@H2C5 2 H@C=5 H96C6 px DF>>2C:K6D D4:6?E:7:4 A2A6CD 2?5 >2?286D A6@A=6’D D4965F=6D[ >2<:?8 DFC6 E92E “AC@323=J” >62?D “AC@323=J” :D 2 G:E2= DE6A :? >2<:?8 E96D6 DJDE6>D C6=:23=6 A2CE?6CD C2E96C E92? ;FDE D@A9:DE:42E65 A2CC@ED]k^Am
kAmk6>m%96 k2 9C67lQ9EEADi^^E964@?G6CD2E:@?]4@>^E@A:4D^C6D62C49\3C:67\gbab`Qm#6D62C49 qC:67k^2m :D 2 D9@CE E2<6 @? :?E6C6DE:?8 24256>:4 H@C<]k^6>mk^Am
kAmk6>m%9:D 2CE:4=6 :D C6AF3=:D965 7C@> k2 9C67lQ9EEAi^^E964@?G6CD2E:@?]4@>^FDnFE>0D@FC46l{66U2>AjFE>0>65:F>lC6=2E65\=:?<U2>AjFE>042>A2:8?lC6=2E65\=:?<\_U2>AjFE>04@?E6?El2CE:4=6\afdeaeQm%96 r@?G6CD2E:@?k^2m[ 2 ?@?AC@7:E[ :?56A6?56?E ?6HD @C82?:K2E:@? 3C:?8:?8 J@F 724ED 2?5 ECFDEH@CE9J 2?2=JD:D E@ 96=A J@F >2<6 D6?D6 @7 @FC 4@>A=6I H@C=5] xE H2D HC:EE6? 3Ji k2 9C67lQ9EEADi^^E964@?G6CD2E:@?]4@>^AC@7:=6D^>2J2?<\<6;C:H2=\`a`b_ahnFE>0D@FC46l{66U2>AjFE>0>65:F>lC6=2E65\=:?<U2>AjFE>042>A2:8?l3J=:?6\=:?<U2>AjFE>04@?E6?El2CE:4=6\afdeaeQm|2J2?< z6;C:H2=k^2m[ k6>mk2 9C67lQ9EEADi^^E964@?G6CD2E:@?]4@>^:?DE:EFE:@?D^F?:G6CD:EJ\@7\D@FE96C?\42=:7@C?:2\`aednFE>0D@FC46l{66U2>AjFE>0>65:F>lC6=2E65\=:?<U2>AjFE>042>A2:8?l3J=:?6\=:?<U2>AjFE>04@?E6?El2CE:4=6\afdeaeQm&?:G6CD:EJ @7 $@FE96C? r2=:7@C?:2k^2mk^6>mk^6>mk^Am
kAmkDEC@?8m#625 >@C6ik^DEC@?8mk^Am
kF=mk=:mk2 9C67lQ9EEAi^^E964@?G6CD2E:@?]4@>^5@?E\36E\H:E9\492E8AE\DEF5J\D9@HD\=2?8F286\2:D\@7E6?\>2<6\:CC2E:@?2=\564:D:@?D\a_ahbenFE>0D@FC46l{66U2>AjFE>0>65:F>lC6=2E65\=:?<U2>AjFE>042>A2:8?lC6=2E65\=:?<_U2>AjFE>04@?E6?El2CE:4=6\afdeaeQms@?’E 36E H:E9 r92Ev!% – DEF5J D9@HD =2?8F286 pxD @7E6? >2<6 :CC2E:@?2= 564:D:@?Dk^2mk^=:mk=:mk2 9C67lQ9EEAi^^E964@?G6CD2E:@?]4@>^:E\E2<6D\2\3@5J\E@\F?56CDE2?5\E96\H@C=5\H9J\492E8AE\2?5\@E96C\=2?8F286\2:D\5@?E\<?@H\H92E\E96JC6\D2J:?8\a_`ag_nFE>0D@FC46l{66U2>AjFE>0>65:F>lC6=2E65\=:?<U2>AjFE>042>A2:8?lC6=2E65\=:?<`U2>AjFE>04@?E6?El2CE:4=6\afdeaeQmxE E2<6D 2 3@5J E@ F?56CDE2?5 E96 H@C=5 – H9J r92Ev!% 2?5 @E96C =2?8F286 pxD 5@?’E <?@H H92E E96J’C6 D2J:?8k^2mk^=:mk=:mk2 9C67lQ9EEAi^^E964@?G6CD2E:@?]4@>^2CE:7:4:2=\>6E24@8?:E:@?\8:G:?8\2?\2:\E96\23:=:EJ\E@\E9:?<\23@FE\:ED\E9:?<:?8\af__aenFE>0D@FC46l{66U2>AjFE>0>65:F>lC6=2E65\=:?<U2>AjFE>042>A2:8?lC6=2E65\=:?<aU2>AjFE>04@?E6?El2CE:4=6\afdeaeQmpCE:7:4:2= >6E24@8?:E:@?i v:G:?8 2? px E96 23:=:EJ E@ ‘E9:?<’ 23@FE :ED ‘E9:?<:?8’k^2mk^=:mk^F=m
kA 4=2DDlQ7:?6\AC:?EQmk6>mkDA2?m|2J2?< z6;C:H2= C646:G6D 7F?5:?8 7C@> E96 s676?D6 p5G2?465 #6D62C49 !C@;64ED p86?4J 2?5 E96 }2E:@?2= x?DE:EFE6D @7 w62=E9]k^DA2?mk^6>mk^Am
kAmk:>8 DC4lQ9EEADi^^4@F?E6C]E964@?G6CD2E:@?]4@>^4@?E6?E^afdeae^4@F?E]8:7n5:DEC:3FE@Cl=66Q DEJ=6lQ96:89Ei`AIjH:5E9i`AIjQ 2=ElQQ ^mk^Am
(0) comments
Welcome to the discussion.
Log In
Keep it Clean. Please avoid obscene, vulgar, lewd, racist or sexually-oriented language.
PLEASE TURN OFF YOUR CAPS LOCK.
Don't Threaten. Threats of harming another person will not be tolerated.
Be Truthful. Don't knowingly lie about anyone or anything.
Be Nice. No racism, sexism or any sort of -ism that is degrading to another person.
Be Proactive. Use the 'Report' link on each comment to let us know of abusive posts.
Share with Us. We'd love to hear eyewitness accounts, the history behind an article.