Scripting Helpers is winding down operations and is now read-only. More info→
Ad
Log in to vote
1

Datastore error is not getting fixed, thoughts?

Asked by 3 years ago
Edited 3 years ago

I am storing block data and have come to a limit with this error: 502: API Services rejected request with error. Value length exceeded the allowed maximum. Value length was 272914 UTF8 bytes. Maximum allowed length is 262144 UTF8 bytes.

This is my script, I have tried limiting so many models to being stored in one datastore, still does not work.

local DSS = game:GetService("DataStoreService")
local modelDataStore = DSS:GetDataStore("e")
local modelDataStore2 = DSS:GetDataStore("e")
sniff = 1

game.ReplicatedStorage.Save.OnServerEvent:Connect(function(player)
    game.Workspace.Check.Value = "Saving.."
    local models = {}
    for _, model in pairs(game.Workspace.Blocks:GetChildren()) do
        local modelData = {}        
        local primaryPart = model.PrimaryPart
        local position = model.PrimaryPart.Position
        local direction = primaryPart.CFrame.lookVector
        modelData.ModelName = model.Name
        modelData.Position = {position.X, position.Y, position.Z, direction.X, direction.Y, direction.Z}
        if #models > 999 then

            sniff = 2
            models[#models + 1] = modelData
            print(models.ModelName) 
        else            
            print(100000*99999999999999)
            sniff = 1
            models[#models + 1] = modelData
            print(models.ModelName) 

    end end
    if sniff == 1 then
        modelDataStore:SetAsync(player.UserId, models)
    elseif sniff == 2 then
        wait(1)
        modelDataStore2:SetAsync(player.UserId, models)
    end
    print('done')
end)

thoughts? (that is my saving script, lmk if u need my loading script for some reason) Thanks, Jail.

(loading)

local DSS = game:GetService("DataStoreService")
local modelDataStore = DSS:GetDataStore("WelpWeAreAllDead")
local modelDataStore2 = DSS:GetDataStore("WelpWeAreAllDead2")

game.Players.PlayerAdded:Connect(function(player)

    game.Workspace.plr.Value = player.Name
    local models = modelDataStore:GetAsync(player.UserId)
    local models2 = modelDataStore2:GetAsync(player.UserId.."_")
    game.Workspace.Blocks:ClearAllChildren()

    for _, modelData in pairs(models) do
        local targetModel = modelData.ModelName
        local information = modelData.Position
            local variable = game.ReplicatedStorage:FindFirstChild(targetModel)
            print(variable)
        local newModel = variable:Clone()
        local position = Vector3.new(information[1], information[2], information[3])
        local lookVector = Vector3.new(information[4], information[5], information[6])
        newModel:SetPrimaryPartCFrame(CFrame.new(position, position + (lookVector * 5)))
            newModel.Parent = game.Workspace.Blocks
    end
    for _, modelData2 in pairs(models2) do
        local targetModel2 = modelData2.ModelName
        local information2 = modelData2.Position
            local variable2 = game.ReplicatedStorage:FindFirstChild(targetModel2)
            print(variable2)
        local newModel2 = variable2:Clone()
        local position2 = Vector3.new(information2[1], information2[2], information2[3])
        local lookVector2 = Vector3.new(information2[4], information2[5], information2[6])
        newModel2:SetPrimaryPartCFrame(CFrame.new(position2, position2 + (lookVector2 * 5)))
            newModel2.Parent = game.Workspace.Blocks
end
end)

3 answers

Log in to vote
2
Answered by 3 years ago
Edited 3 years ago

You have these options:

  1. Use HttpService:JSONEncode to determine how long the data is before attempting to SetAsync it. If it exceeds the the maximum length, you can...
    • Tell the user they have too much stuff
    • Prevent the user from getting that much stuff in the first place (means you'd have to check this when they're creating more models)
    • Split up the data into two or more keys (I think you're trying to do this, but you're saving the same list of models to the same data store and same key each time)
  2. Save less data (ex, do you need to save all the blocks in the workspace, or only the ones belonging to the player?)
  3. Use a more efficient save format

Let's explore splitting up the data. Your script can be fixed up a bit:

  • ipairs should always be used instead of pairs when iterating over lists (such as what you get from GetChildren)
  • Use only one data store, but multiple keys
  • Make sure each key gets a different list
local DSS = game:GetService("DataStoreService")
local modelDataStore = DSS:GetDataStore("e")

local function getModelData(model)
    local modelData = {}
    local primaryPart = model.PrimaryPart
    local position = model.PrimaryPart.Position
    local direction = primaryPart.CFrame.lookVector
    modelData.ModelName = model.Name
    modelData.Position = {position.X, position.Y, position.Z, direction.X, direction.Y, direction.Z}
    return modelData
end
game.ReplicatedStorage.Save.OnServerEvent:Connect(function(player)
    game.Workspace.Check.Value = "Saving.." -- I assume this line is for debugging?
    local allLists = {} -- list of models
    local models = {}
    allLists[1] = models
    for _, model in ipairs(game.Workspace.Blocks:GetChildren()) do
        if #models > 999 then -- Start a new list
            models = {}
            allLists[#allLists + 1] = models -- record the new list in allLists
        end
        models[#models + 1] = getModelData(model)
    end
    for i, models in ipairs(allLists) do
        modelDataStore:SetAsync(player.UserId .. "_" .. i, models)
    end
    print('done')
end)

(Note: I created the 'getModelData' function for organization, but there's nothing wrong with leaving it how it was.)

If you want it to be compatible with previous data, change that SetAsync line to:

        modelDataStore:SetAsync(i == 1 and player.UserId or player.UserId .. "_" .. i, models)

For example, if the user id is 1234, then the first 1000 models will be saved in 1234, the next 1000 in 1234_2, then 1234_3, and so on.

Note that you should use a number that is as large as you can make it (not just 1000), using the worst case for each model (ex get a number with as many decimals as possible for all dimensions).

When loading, you need to know when to stop loading in more information. You'll have to decide how you want to do that. For example, you could store in the first data store key how many keys were used to save it all.

For #2, I'll assume you want all the parts in the workspace, but have you considered storing only 1 or 2 decimals worth of information (since the user probably doesn't care about the difference between 56.13 versus 56.134)?

[Edit] I just discovered that JSON will encode 1.01 as 1.010000000000000008881784197! Therefore, it is critical for you to decide how many decimals you want and to multiply your numbers appropriately. For example, say you believe that 1 decimal will do. Then you should perform math.floor(num*10+0.5) to get a number that JSON won't explode in length (ex, it'll transform 1.234 into 12 and 1.85 into 19), and then when you load you just divide the number by 10 to get something close enough to the original (12 / 10 = 1.2; 19 / 10 = 1.9).

For #3, you can theoretically convert all your data into a string. The principle is this: if you have an integer that could be from 0 to 255, then you could store that integer in a single character using string.char(num). Unfortunately, JSON won't tolerate you using string bytes 128 through 255 unless they're in a particular format, and there are 34 other characters that JSON will escape in some way (so that instead of storing a single character, it may store 2 or even 6!) You can get a list of which ones it escapes here: https://devforum.roblox.com/t/text-compression/163637/5 We're left with 94 characters that are usable and don't get expanded into multiple characters. The end result is a system that still packs information more tightly than what you're doing (since 94 possibilities per character is a lot better than the ~10 you get normally when storing numbers - with just two characters you can store a number from 0 to 8835 instead of 0-99), but this requires some math to perform, can be very difficult to debug, and takes the server extra processing time to calculate.

That link above also leads to a compression algorithm someone wrote which you can also just apply without doing any of the math yourself - but if you use it, you can't know in advance how many models will fit in a particular key (so I'm not sure if this would help you more than it'd open you up for bugs), so relying on it and hoping that it'll all fit in a single key is not safe. What you could do is to use HttpService:JSONEncode on your entire list of models, use the text compression algorithm on the result, and then break up that string into multiple pieces and store them in separate data store keys.

0
This is amazing, just a lot to take in, mind helping me make it work with my loading script? (I edited it in on the question) JailBreaker_13 350 — 3y
Ad
Log in to vote
0
Answered by
Mroczusek 111
3 years ago

I think that you've unfortunately reached the limits of datastore if you're getting that error message.

0
There is always data to compress or something in some way. I just dont know how. JailBreaker_13 350 — 3y
Log in to vote
0
Answered by 3 years ago

Have you enabled API in your game?? It won't work if you haven't enabled API.

0
Of course i have, the data is just too big to be saved. JailBreaker_13 350 — 3y
0
alright rayhansatria 142 — 3y

Answer this question