Don't rely on AI coding.
Mar. 11th, 2026 05:35 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
estiroseMy audience probably isn't much into having AI help code software, but I ran an experiment just out curiosity. Basically, I asked 3 different AIs (ChatGPT, Claude, and Llama) code a Stardew Valley Content Patcher mod. I am very familiar with coding these things and can easily spot errors. None of them got the coding right.
The mod request was simple: create the code for a machine that would output 1 random item per day, acting similar to the Worm Bin and/or the Statue of Endless Fortune. (These actually work slightly different in their coding despite essentially doing the same output at start of day thing.) Most of the time, the player should get common farm items - stone, wood, fiber. But upon occasion, the player should be able to get either the Dish of the Day, a random seasonal vanilla item, or a random object in one of a couple of categories.
(It was fun to play with, but I initially put the percentages up too high for the random items and so I was getting stuff that was way out of balance with what I really wanted. Not that I really needed this; I just like random generators!)
All of them were able to write a good manifest.json, which is kind of the introductory file that all Stardew Valley mods use.
ChatGPT 5.2: Mostly decent code, including finding DISH_OF_THE_DAY and RANDOM_BASE_SEASON_ITEM (the saloon's Dish of the Day and the pool for when you break mine barrels/fish in garbage cans respectively) but refused to recognize that you could concisely put in percentages in; it insisted on multiple entries for wood/stone/fiber to weigh them higher.
Claude Sonnet 4.5: Pulled a similar odd way of weighing choices, though when I gave it an example from one of my other mods, it did fix the code. However, it failed to recognize RANDOM_BASE_SEASON_ITEM when I gave it the example of "trash/barrels"; it created a list of the trash items, which works, but not really what I wanted. It did write a nice recipe entry to make the machine though. (Mine I just set to buy from Robin.)
Llama 4 Maverick: I have no idea what in the heck this was doing. It got the structure right, but not much else.
ETA: I forgot Gemini!
Google Gemini 3: Gemini, like Claude, failed to find RANDOM_BASE_SEASON_ITEM; it recognized the idea of random chances and mostly got the idea correct but had the code for them wrong (see below on how to write that in Content Patcher). Even when I linked it to the wiki page that included that, it insisted I was wrong.
Here's the meat of the mod:
"OutputItem": [
{
"CustomData": null,
"ItemId": "(O)388",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 3,
"Quality": 0,
"ModData": null,
"
},
The section above is my default. Apparently I decided that if nothing else applied, that I would get 1-3 wood. (O)388 is the Item ID for wood.
{
"CustomData": null,
"ItemId": "DISH_OF_THE_DAY",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 1,
"Quality": 0,
"ModData": null,
"Condition": "RANDOM 0.1",
},
This section tells the game that there's a 10% chance that the machine will give me the Dish of the Day instead.
{
"CustomData": null,
"ItemId": "RANDOM_BASE_SEASON_ITEM",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 1,
"Quality": 0,
"ModData": null,
"Condition": "RANDOM 0.1",
},
This section tells the game that there's a 10% chance that the machine will give me a random item from the list of base season items (that is, the ones you get from breaking barrels and digging through trash cans).
{
"CustomData": null,
"ItemId": "RANDOM_ITEMS (O)",
"PerItemCondition": "ITEM_CATEGORY Target -2 -5 -6 -12 -15 -16 -20 -21",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 1,
"Quality": 0,
"ModData": null,
"Condition": "RANDOM 0.1",
}
This is another 10% chance. This one tells the game to choose an item out of one of 8 categories: gems, eggs, milk, minerals, metal resources, building resources, junk, or bait.
Assuming I've coded this correctly - still testing it, wouldn't put it in a serious game - it would choose an item for any categories where this was random (that is, not the dish of the day or the wood), then check to see if any of them rolled under 10%. It then chooses one category randomly and spawns the item. The order might be slightly different (the game might see if the percentage applies first, then rolls the random item). Not sure, doesn't change things.
The mod request was simple: create the code for a machine that would output 1 random item per day, acting similar to the Worm Bin and/or the Statue of Endless Fortune. (These actually work slightly different in their coding despite essentially doing the same output at start of day thing.) Most of the time, the player should get common farm items - stone, wood, fiber. But upon occasion, the player should be able to get either the Dish of the Day, a random seasonal vanilla item, or a random object in one of a couple of categories.
(It was fun to play with, but I initially put the percentages up too high for the random items and so I was getting stuff that was way out of balance with what I really wanted. Not that I really needed this; I just like random generators!)
All of them were able to write a good manifest.json, which is kind of the introductory file that all Stardew Valley mods use.
ChatGPT 5.2: Mostly decent code, including finding DISH_OF_THE_DAY and RANDOM_BASE_SEASON_ITEM (the saloon's Dish of the Day and the pool for when you break mine barrels/fish in garbage cans respectively) but refused to recognize that you could concisely put in percentages in; it insisted on multiple entries for wood/stone/fiber to weigh them higher.
Claude Sonnet 4.5: Pulled a similar odd way of weighing choices, though when I gave it an example from one of my other mods, it did fix the code. However, it failed to recognize RANDOM_BASE_SEASON_ITEM when I gave it the example of "trash/barrels"; it created a list of the trash items, which works, but not really what I wanted. It did write a nice recipe entry to make the machine though. (Mine I just set to buy from Robin.)
Llama 4 Maverick: I have no idea what in the heck this was doing. It got the structure right, but not much else.
ETA: I forgot Gemini!
Google Gemini 3: Gemini, like Claude, failed to find RANDOM_BASE_SEASON_ITEM; it recognized the idea of random chances and mostly got the idea correct but had the code for them wrong (see below on how to write that in Content Patcher). Even when I linked it to the wiki page that included that, it insisted I was wrong.
Here's the meat of the mod:
"OutputItem": [
{
"CustomData": null,
"ItemId": "(O)388",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 3,
"Quality": 0,
"ModData": null,
"
},
The section above is my default. Apparently I decided that if nothing else applied, that I would get 1-3 wood. (O)388 is the Item ID for wood.
{
"CustomData": null,
"ItemId": "DISH_OF_THE_DAY",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 1,
"Quality": 0,
"ModData": null,
"Condition": "RANDOM 0.1",
},
This section tells the game that there's a 10% chance that the machine will give me the Dish of the Day instead.
{
"CustomData": null,
"ItemId": "RANDOM_BASE_SEASON_ITEM",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 1,
"Quality": 0,
"ModData": null,
"Condition": "RANDOM 0.1",
},
This section tells the game that there's a 10% chance that the machine will give me a random item from the list of base season items (that is, the ones you get from breaking barrels and digging through trash cans).
{
"CustomData": null,
"ItemId": "RANDOM_ITEMS (O)",
"PerItemCondition": "ITEM_CATEGORY Target -2 -5 -6 -12 -15 -16 -20 -21",
"MaxItems": null,
"MinStack": 1,
"MaxStack": 1,
"Quality": 0,
"ModData": null,
"Condition": "RANDOM 0.1",
}
This is another 10% chance. This one tells the game to choose an item out of one of 8 categories: gems, eggs, milk, minerals, metal resources, building resources, junk, or bait.
Assuming I've coded this correctly - still testing it, wouldn't put it in a serious game - it would choose an item for any categories where this was random (that is, not the dish of the day or the wood), then check to see if any of them rolled under 10%. It then chooses one category randomly and spawns the item. The order might be slightly different (the game might see if the percentage applies first, then rolls the random item). Not sure, doesn't change things.
