How do I change the color of a frame with a script?

You asked: How do I change the color of a frame with a script?

This is actually quite simple and you were very close. Color3.new() takes 3 arguments. The first argument, r, stands for red and is the red value. The second argument, g, stands for green and is the green value. The final argument, b, stands for blue and is the blue value. Every argument must be a number between 0 and 1. You can't actually use integers for the values (besides 0 and 1). So, most people usually do:

Color3.new(myRgbRValue/255, myRgbGValue/255, myRgbBValue/255);

Diving by 255 will actually give you a number between 0 and 1. Although there's an alternative. You can just do Color3.fromRGB(Red, Green, Blue). This is much simpler to use and doesn't require any math.

In conclusion, your fixed code should look like this:

script.Parent.BackgroundColor3 = Color3.fromRGB(math.random(1,256),math.random(1,256),math.random(1,256));

Here's the wiki's documentation on Color3.new() in case you want it:

"Description: Creates a Color3 whose values are (0,0,0) [black]

Color3 Color3.new(number r, number g, number b)"