Working on hard problems

This is all pretty much personal preference and very biased to the computer science field.

What is an easy problem?

I just saw a X post about creating an app for swiping left or right for clothing for feedback? Is this interesting? Will it leave a lasting impression? Is it cool?

It does make money. What makes it interesting is not the technical part, but the psychological attribute. What makes a person click or swipe? Does the culture matter? Are they apple or android users? Etc.

Is this an easy problem? Technically, it kind of sounds like it is. Scaling might be hard (thousands, millions, billions) of users? Analysis of TBs of data is hard (I think)

So, what are hard problems?

Problems you may find interesting to you and you don’t have enough knowledge in and believe others don’t have knowledge either. Example is world hunger. Hard problems take a long time to work on. Varies by how long, depends on field. But what I mean by long is that it takes a lot of subconscious plus conscious (by default) effort to do so. You have take action and get feedback, repeat and think.

Hard problems take a long time to work on.

Is this true? I feel like yes. Most resources are spent discovering and iterating until it may become an easy problem.

Are hard problems money problems?

Typically yes. Very few fields (like math) don’t touch the real world. Money fuels problems and vice versa. More money sometimes gives more incentive (tied to life) and puts a number on progress. however, warn that money should not always be the top motivator. False sense of accomplishment when money comes in or unstable if money doesn’t come in. Money should be placed as the same level or less than another motivator

Are interesting problems the same as hard problems?

v1 of thought

There’s a difference between hard and interesting problems. Sometimes interesting problems are made up - it’s because you don’t know the field well enough and make a lot of assumptions. An example is in AI, where people want to merge say systems + ai, but don’t know how the ai flow works. For example, how does data get transformed from a piece of text to an image. What algorithms are required to do this and what computational limits are there? Are there alternatives? Etc. Or AI + coding, do people know how coding workflows are like? Is it different for webdev vs ml vs systems folks? Does the IDE matter? How does the AI interact? Voice? Chat suggestions? Why not just copy and paste into browser? Does it save half a second? This matters?

But interesting problems are generally fun problems to work on, I think.

v2 of thought (1 month later)

No, though interesting problems can easily give birth to hard problems. Define interesting - some problem you are excited about but somewhat know it isn’t hard or don’t care if it’s hard/easy. Llms are interesting, let’s learn! Hard problems might be training it. How to validate hard problems? Ask people, generally people say think or do. Ask them what they think about your problem, rather than to ask them for a solution. See if they can answer your problem confidently or correctly (hard)

What is learning / looking for problems?

Learn to find problems to solve. That’s hard. You don’t know what you’re getting yourself into. And the question is: How quick can I find this problem? Immediate thought it luck + amount of effort, but there’s ways to increase your chances. That’s for another topic though. The question I’m looking here is how to find a hard problem. And could be interesting.

Let’s say you’re into AI and want to solve some intersection between ai and whatever you’re interested in. For example, you do coaching in spefically soccer with ai vs coding with ai, another example folding clothes with ai vs replacing the entire food industry with robots.

Each of these has varying degrees of hardness and interest and can be broken down.

In computer science, this could be tiered

AI assistant for coding?

1) optimize a specific program for a specific workload 2) optimize a specific program for a class of programs 3) optimize a specific program for general programs

1) Python 2) C++, Python, java, all the common languages 3) For any language, even english

Reading some articles and confirming the beliefs of hard problems (After reading the articles)

An Interview with AMD CEO Lisa Su About Solving Hard Problems

When I went to MIT, at that time it was a decision between electrical engineering and computer science. Computer science, you could just write software programs, whereas electrical engineering, you had to build things. I wanted to build things. What I learned actually was one of the very first products that I worked on was a microprocessor with copper interconnects, and it turns out that the last 5% of what it takes to get a product out is probably the hardest, where most of the secret sauce is. And if you learn how to do that, then frankly, it’s— Hard in producing something. That’s the hard part Yes, as a young researcher you think, “Hey, I’m ready to move on to my next thing”, and you realize that it’s so rewarding to see your product actually ship and go on the shelf and you can walk into Best Buy and buy it. Those are the types of things that I learned.

Hard problems evolve over time. Maybe another hard problem is more interesting. Or it may be even a harder problem

Just out of curiosity, how deeply do you need to be involved in things like, not you specifically, but AMD generally, now that you’re fabless, in thinking about that actual final mile? What’s the degree of interaction with, say, a TSMC or your packaging partners or whatever it might be, and actually getting the yields up and out the door? LS: It’s definitely true for us as a fabless company or a design company. We are actually doing end-to-end development, so you can imagine from the day one of concept of a product — actually even before that, we’re thinking about what technologies are going to be ready, what are the next big things that we should bet on? That goes all the way through. Sometimes it could be a five-year cycle or even longer before the technology actually comes to fruition. We’re right there at the end as well, ensuring that it ships with high quality, at the right yields, right cost structure, in high volume production Think about problems usual people never think about LS: Across multiple companies, yes. What I would say honestly is, these are decisions that are made that are more around architectural decisions. From that standpoint, whether you talk about any of the PlayStation consoles or some of the other work that we’ve done in partnership — we being AMD, but it was similar during that time at IBM — it really is a close collaboration of what the customer or partner is trying to achieve. The Cell processor was extremely ambitious at that time, thinking about the type of parallelism that it was trying to get out there. Again, I would say from a business standpoint, it was certainly successful. As you rank things, I think history will tell you that there may be different rankings.

Think about business, research instead

LS: I have always straddled the technology/business line there. At Freescale I actually started as CTO. I joined as CTO, and then over a couple of years I ended up running the networking and a multimedia business, that was definitely a choice and the choice is, at the end of the day, I want to drive outcomes, and to drive outcomes requires, yes, the technology is great, but you need to have the right business strategy.

Your perspective of what is a hard problem changes over time - it may become an easy problem. Or uninteresting. You could find an even harder problem.

Is that the limiter for a lot of technical people, is they under-appreciate all the drivers of outcomes that have nothing to do with technology? LS: I think that that is something that technologists have to learn. And by the way, there are phenomenal CTOs who truly understand that. My CTO right now, Mark Papermaster, he was my partner in crime at IBM, we grew up together, and then we’ve been partners here at AMD, he truly understands that technology is great, but you also need to drive business outcomes. That’s what I love about what I get to do, because yes, I get to put together great tech with a phenomenal team, but there’s also an opportunity to drive very significant business outcomes. This is a biased view. However it does show that her hard problems have changed after seeing the outcomes LS: The best way I say is there are a couple of principles. First, the fact is the world needs more semiconductors. Semiconductors, chips are now foundational to so much of what we do and much of what we do are, let’s call it standard products that fit a broad set of use cases. But you do find those high volume applications like game consoles, like some of the work that’s being done in the cloud right now, like some of the AI work I believe will be customized, and in these cases, because the volume is so high, it makes sense to customize. That’s something that I’ve always believed. It’s part of our strategy, it’s part of our strategy of deep partnerships. So if you have the right building blocks, then you can work with the broad set of customers to really figure out what they need to accomplish their vision.

She thinks that semiconductors are general and foundational. I agree, it’s a hard problem, since not many people have figured it out

LS: I think the important thing is looking at which markets really lend themselves to let’s call it significant customization, and it’s not everything. Probably your IoT device, you’re not going to want to do that because it just won’t return on investment. But for large computing capabilities, I think the combination of the right IP plus the ability to work deeply with partners. By the way, I should say, it doesn’t all have to be hardware customization, there’s a lot that we can do in software work as well, I think it is one of the important trends going forward.

She thinks about all layers of the stack from customers all the say down to the chip itself. This is important. Not many do this when thinking about problems

So from a strategy standpoint, I think we had some choices to make. If you remember back, this is 2014, the exciting thing then was mobile phones, like apps processors. So we would have these conversations like, “Should we go into phones?”, and we were like, “No, we shouldn’t because we’re not a phone company. There are others who are much better at that, we are a high performance company, so we have to build a roadmap that leverages our strengths and that requires us to revamp the way we do architecture and design and manufacturing”. I know how to do that, it’ll take time, you can’t do that in 12 months, I think I felt it would take five years. It would take five years, but it was very clear that we had the pieces, we just had to really methodically build that execution engine.

Shows that at this point, she understood what she kind of knew about (that it would take that long), some people take 5 years

So I remember very well spending time with Mark and our architects and trying to decide, “Is this the time that we go to chiplets? Is this the time we’re going to bet the company on going to chiplets?” And we said, “Yes, it is because we’re going to get to much higher performance, many more cores, as well as a much better cost point”, and it gave us tremendous flexibility, and we learned a lot along the way.

Hard problems sometimes comes with time constraints and requires betting

Yeah, it’s a great point. When you think about high-performance computing and just how things have changed, the fact is, the hyperscalers are such a significant piece of the overall market that we have spent a lot of time there, and the point that you make is absolutely true, which is — you’d like to think in every market, the product always wins but that’s not necessarily true. In the hyperscaler market, the best product wins. Yes. LS: And we were able to show that. Frankly, the key thing in this market is, it’s not enough to win once and it’s not enough to win temporally.

Do hard problems require continual iterations? Are they solvable? I feel this is true

It was the next big opportunity, we knew it was the next big opportunity. It was something that Mark and I discussed, which was, by putting CPUs and GPUs together in systems and designing them together, we’re going to get a better answer and the first near-term applications were around super-computing. We were very focused on these large machines that would reside at national laboratories and deep research facilities and we knew that we could build these massively parallel GPU machines to do that. The AI portion, we always also thought about it as clearly a HPC plus AI play.

Interesting view to tackle a hard problem. People with hard problems typically have strong convictions about things. (even if they’re wrong)

I would say it this way. When you look at what the market will look like five years from now, what I see is a world where you have multiple solutions. I’m not a believer in one-size-fits-all, and from that standpoint, the beauty of open and modular is that you are able to, I don’t want to use the word customize here because they may not all be custom, but you are able to tailor.

There are multiple ways to arrive at a “solution” to a hard problem. Some are clearly better.

absolutely believe that it’ll be across the industry. Things like PyTorch, I think PyTorch is extremely widely adopted, OpenAI Triton, similar. These are larger industry things where frankly, part of the desire is it takes a long time to program down to the hardware. Everyone wants to innovate quickly, and so the abstraction layer is good from the standpoint of just rapid innovation.

Abstractions… are the hard problems underneath?

This is the next arc. I can tell you that the thing that’s so amazing about where we are today in high-performance computing is, who would imagine? It’s like a new world. It’s an incredibly exciting.

Motivation.