Anyway comical understanding of the two extreme sides of 40k aside, there are generally two distinct build categories - full on spam where nothing is a singular and everything is repeated and rainbow/battleforce armies where you've got a little bit of everything. These are very broad strokes so let's not nitpickle. Ironically a lot of people fall somewhere bang in the middle as a compromise between pure optimisation and personal feel. What this post is going to look at is the key concepts of both sides of the argument and then look at making a Rainbow army which is actually semi-decent on the tabletop by applying good list building skills.
Personally full-on spam armies are very restrictive and whilst they offer the height of optimisation in terms of whatever that list is trying to do (i.e. old Immo-spam, RazorKnights, RazorWolves, Flamestorm BA, etc.), they are generally gaining that optimisation at the cost of something on the battlefield (i.e. ranged or combat prowess). They do present your opponent with target priority issues, have redundancy to the hilt and are certainly good lists but aren't as 1, 2, 3 LIST and then point and click as people like to make out and rarely will you actually see such lists (i.e. people commonly like to call Hybrid Wolves RazorWolves when there are generally 3-5 Razorbacks).
On the other hand, Rainbow or Battleforce armies are often unfocused, have minimal redundancy and are often literally whatever a person has available to them or what they like. Nothing wrong with this what so ever, unless you think it's competitive and try to tell people so. That being said, can one make a decently competitive list without spamming? I know some people such as BroLo hate any sort of spam for some odd aesthetic reason (I ask him to do every medical procedure different, let's see how he likes that!) so let's see if we can't make a list which is decently competitive without any replications. This is possible outside of spamming because we know certain units fulfil similar roles - spamming is not the only way of redundancy. For example, if I wanted medium to high strength long ranged firepower from a Marine list I have a lot of options. Devastators with Missile Launchers, Land Speeder Typhoons, Rifledreads or Razorbacks are all great options which provide this. If I took one of each I've got four ranged units each with a different set of advantages and disadvantages. I still have the ability to reach out and hurt four different targets a turn but each unit is going to be a little better or a little worse depending upon the situation. Doing this stops me from gaining maximum efficiency in certain situations but also stops my army 'falling off a cliff' when it comes up against it's weakness.
This is a major principle of good list design and if you take it just a bit further and allow yourself to replicate something twice (i.e. 2x Rifledreads) whilst still covering your redundancy with other units, you'll get an army list which is flexible but has excellent redundancy. This can be one of the issues spamming lists have - they lose a bit of flexibility. Rainbow lists on the other hand generally don't have that redundancy built into their list and opposing forces can pick out what is most dangerous to them and then walk all over it.
Even with this principle in mind, you cannot simply grab four of these units with similar roles, four of these units with similar roles and four of these units with similar roles - I have an army! This is bad list building and often what you see in such Rainbow/Battleforce armies. Whilst you don't HAVE to spam to get redundancy, you cannot just willy nilly pick out units and put them into an army. There needs to be some sort of cohesion and army 'theme.' Are you going for shooting, midfield or assault based? And what units are you going to use to cover your weaknesses in other areas? These are all important questions you have to ask and when you take duplicates of units, are easier to do as you can use certain Force Organisation Charts for specific tasks (i.e. Grey Knights use Heavy Support to field ranged firepower to support their midfield units).
So let's put this all together and see if we cannot make an effective Rainbow list with no duplications. It's not going to be hyper competitive but it is an excellent exercise in ensuring you have good list building principles without using what some would call the crutch of spamming. I'm going to post the list I have in mind and then we shall discuss it.
Master of the Forge w/Conversion Beamer
Dreadnought w/2x Twin-linked Autocannons
Dreadnought w/twin-linked Lascannon, Missile Launcher
5x Scouts w/Camo Cloaks, Missile Launcher
10x Tacticals w/Missile Launcher, meltagun, combi-melta, LasPlas Razorback
5x Tacticals w/combi-melta, LasPlas Razorback
10x Tacticals w/MM, meltagun, combi-melta, Rhino
2x Land Speeder Typhoons
2x MM/HF Land Speeders
Predator w/Lascannon sponsons
Predator w/Heavy Bolter sponsons
5x Devastators w/4x Missile Launchers, Rhino
1955 points
36 infantry, 12 vehicles
There are some obvious changes we could make if we were looking for a fully competitive list. Change the ACLC Pred back to a Dakka Pred. Change the TLLC/ML Dread back to a Rifledread. Buy a 3rd Rifledread with points saved. Split one of the Speeder squads up. Change the MM to a ML on the 2nd Tactical squad (allowing the 2 ML halves to combat squad, the LasPlas RBacks stay back and shoot, the melta units into the Rhinos). Suddenly with a few duplications we have a much more refined list - still not what many would say is top-notch 100% competitive but quite a good list which covers all the basics. Let's look at that here.
We've got a lot of shooting. Two Dreads, two preds, one combat squad of Marines, two Razorbacks, Devastators, Typhoons and Scouts + MotF. Each of these units has a mildly different role but with some overlap. All the missile based squads can shoot at both tanks and infantry and vary from static (combat squad, Devs, Scouts + MotF) to mobile (Typhoons) with some scoring thrown in (combat squad, Scouts). Some are simply good at dropping transports (Rifledread) whilst others can drop heavier tanks and hurt heavy infantry (ACLC Pred, LasPlas). We also have units which can really generate a lot of wounds on infantry and have some minor duality against tanks (Dakka Pred, Scouts + MotF). We have some aggressive units with the MM/HF Speeders and melta Bunkers both of which can move forward and aggressively engage tanks and weaker infantry. There are still 12 vehicles in the list and a little low on 36 infantry but backed up by Marine statlines. The list however lacks in the combat department and is generally inefficient. The changes suggested above increase the efficiency of the list and allow a combat unit to be bought (TH/SS + Gate Libby instead of 3rd Rifledread + MotF).
In the end what we see is a not too bad list. If this was sent in to 3++ or seen on the battlefield and the opponent asked for tips, only a relative few changes would generally be recommended. This is very different from a battleforce army where nothing fits together and it quite literally is an army built from battleforces (and GW's battleforces are generally crap for actual playing on the tabletop). I think what this shows is with a bit of thinking you can make a no duplicates at all list work but if you give yourself some flexibility to take some replications, you'll find lists find that balance between optimised spam lists and terrible rainbow lists. This is where the vast majority of successful tournament lists fall as generals compromise between the two sides of the coin and fit the army list to their style of play.