How can I simplify the code for a UI animation I made?

A good programming habit is to follow the DRY (Don't Repeat Yourself) principle, this principle helps keep code short and concise. In your case a good improvement would be to create a function which handles the animation of the GUI object.

The function might look like this:

local TweenService = game:GetService("TweenService")

function tweenUI(guiObject)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        -- Tween properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end

After looking at your code I see that you want to have a tween when the mouse enters, as well as a tween when the mouse leaves. The properties of the tween (Size,Position,Rotation etc.) are also different for each guiObject, as to be expected.

So we can create a new function for when the enter animation should play and another for when the leave animation should play. We can also change it so that the properties of the created tween will be an argument of the function.

local TweenService = game:GetService("TweenService")

function tweenEnter(guiObject,properties)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end
function tweenLeave(guiObject,properties)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end

Actually animating the UI objects would look something like this:

local TweenService = game:GetService("TweenService")

function tweenEnter(guiObject,properties)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end
function tweenLeave(guiObject,properties)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end

local one = script.Parent.one
local two = script.Parent.two
local three = script.Parent.three

local enter1 = tweenEnter(one,PROPERTIES)
local enter2 = tweenEnter(two,PROPERTIES)
local enter3 = tweenEnter(three,PROPERTIES)

local leave1 = tweenEnter(one,PROPERTIES)
local leave2 = tweenEnter(two,PROPERTIES)
local leave3 = tweenEnter(three,PROPERTIES)

one.MouseEnter:Connect(function()
    enter1:Play()
end)
one.MouseLeave:Connect(function()
    leave1:Play()
end)

two.MouseEnter:Connect(function()
    enter2:Play()
end)
twp.MouseLeave:Connect(function()
    leave2:Play()
end)

three.MouseEnter:Connect(function()
    enter3:Play()
end)
three.MouseLeave:Connect(function()
    leave3:Play()
end)

The code above works quite well but there's still quite a lot of repetition, and if you were to add more guiObjects that would become quite a hassle.

So what we can do is to create another function which handles the MouseEnter and MouseLeave events of the GUI, and then applies the tweens to it.

The function might look like this:

function handleTweening(guiObject,enterProperties,leaveProperties)
    local enter = tweenEnter(guiObject,enterProperties)
    local leave = tweenLeave(guiObject,leaveProperties)

    guiObject.MouseEnter:Connect(function())
        enter:Play()
    end)
    guiObject.MouseLeave:Connect(function())
        leave:Play()
    end)
end

The function makes it so that you don't have to create new events and tweens for each GUI object.

The final code would look something like this:

local TweenService = game:GetService("TweenService")

function tweenEnter(guiObject,properties)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end
function tweenLeave(guiObject,properties)
    local info = TweenInfo.new(
        -- Tween info
    )
    local properties = {
        properties
    }
    local tween = TweenService:Create(guiObject,info,properties) -- Create the actual tween
    return tween -- return the tween
end

function handleTweening(guiObject,enterProperties,leaveProperties)
    local enter = tweenEnter(guiObject,enterProperties)
    local leave = tweenLeave(guiObject,leaveProperties)

    guiObject.MouseEnter:Connect(function())
        enter:Play()
    end)
    guiObject.MouseLeave:Connect(function())
        leave:Play()
    end)
end
local one = script.Parent.one
local two = script.Parent.two
local three = script.Parent.three

handleTweening(one,ENTERPROPERTIES,LEAVEPROPERTIES)
handleTweening(two,ENTERPROPERTIES,LEAVEPROPERTIES)
handleTweening(three,ENTERPROPERTIES,LEAVEPROPERTIES)

Hello,

It appears your code works and is written pretty well. In terms of simplicity, your code is already very simple, nonetheless, I can attempt to cut down the number of lines and make your code more simplistic.

Firstly, I notice that for each box, we are tweening each to the same size every time, this means that we can create 1 variable containing the desired size and assign it to each tween:

local globalsize = UDim2.new(0.56, 0, 0.11, 0)

local goalSizePosition = {
    Size = globalsize,
    Position = UDim2.new(0.21, 0 ,0.261, 0)
}

local goalSizePosition2 = {
    Size = globalsize,
    Position = UDim2.new(0.21, 0,0.441, 0)
}

local goalSizePosition3 = {
    Size = globalsize,
    Position = UDim2.new(0.21, 0,0.641, 0)
}

This has simplified your code by making one size variable for each tween. This will allow you to change your code faster if you ever want to tweak this value.

Secondly, I noticed that all your tweens use the style Linear. Because you are using the Linear style, we can use different methods to achieve the same effect:

My preferred method of achieving this effect is by using the :Lerp() function. Lerp stands for Linear Interpolation, which means to take two different values, and find all the points between them (a line between the two points). Using this information, we can take a starting position and an ending position, and use all the points in between to smoothly animate a figure moving from a starting position to an ending position.

Lastly, we can use loops to automatically assign each button's functions.

All together, it looks like this:

local frame = script.Parent

local one = frame.one 
local two = frame.two
local three = frame.three

local Shop = script.Parent.Shop

local uis = {one,two,three} --keep all uis in table so we can loop through them
local startposes = {one.Position,two.Position,three.Position} --keep all positions in table so we can loop through them
local startrotation = 0 -- all have same starting rotation
local startsize = one.Size -- assuming all sizes are the same

local endposes = {UDim2.new(0.21, 0 ,0.261, 0),UDim2.new(0.21, 0,0.441, 0),UDim2.new(0.21, 0,0.641, 0)}
local endrotation = 20 --all use same end rotation
local endsize = UDim2.new(0.56, 0, 0.11, 0) -- all use same end size

local animtime = .3 --time for animations to take palce
local animframes = 100 --amount of frames in each animaton

for i, v in pairs(uis) do --loop through our buttons
    v.MouseEnter:Connect(function() --create function for button v
        for n = 1, animframes do
            v.Position:Lerp(endposes[i],n/animframes) --n/animframes% from start to end pos
            v.Size:Lerp(endsize,n/animframes)--n/animframes% from start to end size

            v.Rotation = startrotation + ((endrotation - startrotation) * (n/animframes)) --roblox's built in linear interpolation function does not work for single numbers, so I made my own version here
            wait(animtime/animframes)
        end
    end)

    v.MouseLeave:Connect(function()
        for n = 1, animframes do
            v.Position:Lerp(startposes[i],n/animframes) --n/animframes% from start to end pos 
            v.Size:Lerp(startsize,n/animframes) --n/animframes% from start to end size

            v.Rotation = endrotation - ((endrotation - startrotation) * (n/animframes))
            wait(animtime/animframes) --roblox's built in linear interpolation function does not work for single numbers, so I made my own version here
        end
    end)
end -- completed all functions for small buttons, continue with rest of code

local tweenInfoShop = TweenInfo.new(
    0.3, 
    Enum.EasingStyle.Quad, 
    Enum.EasingDirection.Out, 
    0, 
    false,
    0 
)
local shopNewPos = UDim2.new(-7, 0, -0.001, 0)

local shopOgPos = UDim2.new(1.2, 0, -0.001, 0)

local tweenEnterShop = TweenService:Create(Shop, tweenInfoShop, {Position = shopNewPos})

local tweenLeaveShop = TweenService:Create(Shop, tweenInfoShop, {Position = shopOgPos})

--Shop

local isShopOpen = false

one.MouseButton1Up:Connect(function()
    if isShopOpen == false then
        tweenEnterShop:Play()
    else 
        tweenLeaveShop:Play()
    end

    isShopOpen = not isShopOpen --micro-optimization, allows you to toggle from false to true easily, saves 1 line
end)

This code takes the number of lines from 138, down to just 70, and achieves the same effect.

Inside the code block, there are annotations describing every change. Below are resources if you'd like to learn more about the concepts used to give this answer:

What is linear interpolation

Roblox's linear interpolation function

Roblox loops

Roblox not function and variables

I hope my answer helped you!

Cheers,

Chase