Scripting Helpers is winding down operations and is now read-only. More info→
Ad
Log in to vote
1

Datastore error is not getting fixed, thoughts?

Asked by 4 years ago
Edited 4 years ago

I am storing block data and have come to a limit with this error: 502: API Services rejected request with error. Value length exceeded the allowed maximum. Value length was 272914 UTF8 bytes. Maximum allowed length is 262144 UTF8 bytes.

This is my script, I have tried limiting so many models to being stored in one datastore, still does not work.

01local DSS = game:GetService("DataStoreService")
02local modelDataStore = DSS:GetDataStore("e")
03local modelDataStore2 = DSS:GetDataStore("e")
04sniff = 1
05 
06game.ReplicatedStorage.Save.OnServerEvent:Connect(function(player)
07    game.Workspace.Check.Value = "Saving.."
08    local models = {}
09    for _, model in pairs(game.Workspace.Blocks:GetChildren()) do
10        local modelData = {}       
11        local primaryPart = model.PrimaryPart
12        local position = model.PrimaryPart.Position
13        local direction = primaryPart.CFrame.lookVector
14        modelData.ModelName = model.Name
15        modelData.Position = {position.X, position.Y, position.Z, direction.X, direction.Y, direction.Z}
View all 35 lines...

thoughts? (that is my saving script, lmk if u need my loading script for some reason) Thanks, Jail.

(loading)

01local DSS = game:GetService("DataStoreService")
02local modelDataStore = DSS:GetDataStore("WelpWeAreAllDead")
03local modelDataStore2 = DSS:GetDataStore("WelpWeAreAllDead2")
04 
05game.Players.PlayerAdded:Connect(function(player)
06 
07    game.Workspace.plr.Value = player.Name
08    local models = modelDataStore:GetAsync(player.UserId)
09    local models2 = modelDataStore2:GetAsync(player.UserId.."_")
10    game.Workspace.Blocks:ClearAllChildren()
11 
12    for _, modelData in pairs(models) do
13        local targetModel = modelData.ModelName
14        local information = modelData.Position
15            local variable = game.ReplicatedStorage:FindFirstChild(targetModel)
View all 34 lines...

3 answers

Log in to vote
2
Answered by 4 years ago
Edited 4 years ago

You have these options:

  1. Use HttpService:JSONEncode to determine how long the data is before attempting to SetAsync it. If it exceeds the the maximum length, you can...
    • Tell the user they have too much stuff
    • Prevent the user from getting that much stuff in the first place (means you'd have to check this when they're creating more models)
    • Split up the data into two or more keys (I think you're trying to do this, but you're saving the same list of models to the same data store and same key each time)
  2. Save less data (ex, do you need to save all the blocks in the workspace, or only the ones belonging to the player?)
  3. Use a more efficient save format

Let's explore splitting up the data. Your script can be fixed up a bit:

  • ipairs should always be used instead of pairs when iterating over lists (such as what you get from GetChildren)
  • Use only one data store, but multiple keys
  • Make sure each key gets a different list
01local DSS = game:GetService("DataStoreService")
02local modelDataStore = DSS:GetDataStore("e")
03 
04local function getModelData(model)
05    local modelData = {}
06    local primaryPart = model.PrimaryPart
07    local position = model.PrimaryPart.Position
08    local direction = primaryPart.CFrame.lookVector
09    modelData.ModelName = model.Name
10    modelData.Position = {position.X, position.Y, position.Z, direction.X, direction.Y, direction.Z}
11    return modelData
12end
13game.ReplicatedStorage.Save.OnServerEvent:Connect(function(player)
14    game.Workspace.Check.Value = "Saving.." -- I assume this line is for debugging?
15    local allLists = {} -- list of models
View all 29 lines...

(Note: I created the 'getModelData' function for organization, but there's nothing wrong with leaving it how it was.)

If you want it to be compatible with previous data, change that SetAsync line to:

1modelDataStore:SetAsync(i == 1 and player.UserId or player.UserId .. "_" .. i, models)

For example, if the user id is 1234, then the first 1000 models will be saved in 1234, the next 1000 in 1234_2, then 1234_3, and so on.

Note that you should use a number that is as large as you can make it (not just 1000), using the worst case for each model (ex get a number with as many decimals as possible for all dimensions).

When loading, you need to know when to stop loading in more information. You'll have to decide how you want to do that. For example, you could store in the first data store key how many keys were used to save it all.

For #2, I'll assume you want all the parts in the workspace, but have you considered storing only 1 or 2 decimals worth of information (since the user probably doesn't care about the difference between 56.13 versus 56.134)?

[Edit] I just discovered that JSON will encode 1.01 as 1.010000000000000008881784197! Therefore, it is critical for you to decide how many decimals you want and to multiply your numbers appropriately. For example, say you believe that 1 decimal will do. Then you should perform math.floor(num*10+0.5) to get a number that JSON won't explode in length (ex, it'll transform 1.234 into 12 and 1.85 into 19), and then when you load you just divide the number by 10 to get something close enough to the original (12 / 10 = 1.2; 19 / 10 = 1.9).

For #3, you can theoretically convert all your data into a string. The principle is this: if you have an integer that could be from 0 to 255, then you could store that integer in a single character using string.char(num). Unfortunately, JSON won't tolerate you using string bytes 128 through 255 unless they're in a particular format, and there are 34 other characters that JSON will escape in some way (so that instead of storing a single character, it may store 2 or even 6!) You can get a list of which ones it escapes here: https://devforum.roblox.com/t/text-compression/163637/5 We're left with 94 characters that are usable and don't get expanded into multiple characters. The end result is a system that still packs information more tightly than what you're doing (since 94 possibilities per character is a lot better than the ~10 you get normally when storing numbers - with just two characters you can store a number from 0 to 8835 instead of 0-99), but this requires some math to perform, can be very difficult to debug, and takes the server extra processing time to calculate.

That link above also leads to a compression algorithm someone wrote which you can also just apply without doing any of the math yourself - but if you use it, you can't know in advance how many models will fit in a particular key (so I'm not sure if this would help you more than it'd open you up for bugs), so relying on it and hoping that it'll all fit in a single key is not safe. What you could do is to use HttpService:JSONEncode on your entire list of models, use the text compression algorithm on the result, and then break up that string into multiple pieces and store them in separate data store keys.

0
This is amazing, just a lot to take in, mind helping me make it work with my loading script? (I edited it in on the question) JailBreaker_13 350 — 4y
Ad
Log in to vote
0
Answered by
Mroczusek 111
4 years ago

I think that you've unfortunately reached the limits of datastore if you're getting that error message.

0
There is always data to compress or something in some way. I just dont know how. JailBreaker_13 350 — 4y
Log in to vote
0
Answered by 4 years ago

Have you enabled API in your game?? It won't work if you haven't enabled API.

0
Of course i have, the data is just too big to be saved. JailBreaker_13 350 — 4y
0
alright rayhansatria 142 — 4y

Answer this question