Virtual/Physical address space unclear why it's needed about book HOT 14 CLOSED

Yea that could work. I find "we just need to do this; don't ask too many questions" very unsatisfying when I read. I like the parts of the book so far that really stress things we need to do because of legacy reasons, but I still feel like it would be better if we had some appendices going into more depth as to why.

For this section, I think it might be appropriate to say something to the effect of "in order to go into full 64-bit mode, we need to set up something called paging. Paging is certain way to implement something called "virtual memory" which will allow the OS to hide the details of the memory layout of the physical machine from programs that run on the computer. The reasons why this is a good idea will come up later in our chapter on 'user spaces'. For now, we'll just take this as something seemingly random we have to do to get into full 64-bit 'long mode'".

Does something like that work? It lets the reader know that this isn't totally random, but that we're not quite ready to fully understand why we need to do this.

from book.

I can try to explain more, and then we can see how best to improve the text in the chapter.

A 64-bit computer can create numbers that are at most 64-bits big. When it comes to addressing memory, that means we are capable of forming (2^64) - 1 or 18,446,744,073,709,551,616 different memory addresses. We can't go higher because any number higher than that would not fit into a 64 bit number.

Machines with 64-bit CPUs, however, always have less actual memory (RAM) than this. My computer for instance has 8 gigabytes of RAM. 8 GB is equal to 8,589,934,592 bytes. So my computer has way less (roughly ~0.0000001%) actual memory slots compared to the number of addresses it can form.

So the problem is that a program written for 64-bit machines cannot know ahead of time how much actual memory a particular machine that will run that program will have. Should it assume 8 GB of RAM?Well my colleague's computer has 16GB of RAM. So this computer will be missing out on half of the memory resources when it runs on his machine!

So the question becomes what happens if I run a program on my machine with 8GB of RAM that accesses memory location 0x200002710? This address is well within the range of valid 64-bit addresses, but it is outside the 8 GB of memory my computer actually has.

One possible way to handle this would be to just crash the program. But it would be nice to be able to run any program built for 64-bit computers. So we solve this with "virtual memory". The OS will make it appear to our programs that they have the full 64-bit address space, but behind the scenes it will convert these addresses to the range our physical computer understands (even sometimes using tricks like writing RAM to our hard drive to give us some more room).

"Paging" is the strategy we will use to map from all these 64-bit addresses to addresses the physical computer actually has. The software that runs on top of the OS like your internet browser or computer game can just assume it can access memory anywhere in that full 64-bit address range, and the OS will make sure that the memory gets mapped to real physical memory.

Does this make sense?

from book.

Better, however why would a program want to access a memory address way outside its physical memory?

from book.

You can run a program that needs 2GiB ram on a computer that only has 1GiB ram. The trick is the so-called swapping, which backups a physical page to the hard disk. Then we can reuse that physical page and map it to some other virtual page. It's slower, of course, but at least it works.

Another reason for virtual memory is that a computer runs multiple programs simultaneously. These programs might read and write to the same addresses. With virtual memory, we can just map these virtual addresses to different physical addresses. So each program runs in its own address space and can't accidentally interfere with other programs.

from book.

@LaylConway A program would want to access memory addresses outside of its physical address space so that it has more addresses to use.

from book.

I may write a program that I want to be able to run on many different computers. I may not know how much physical memory your computer has or how much my bother's computer has. With this set up, the program can assume that they have access to all addresses you can create with a 64-bit number. The operating system takes care of making sure that the physical memory I use is mapped to a valid place in memory.

As @phil-opp points out, using physical memory is made even more complicated when there are several programs running on your computer at once (e.g. a web browser, a video player and calculator). If the OS did not help all of these computers would have to worry about writing over each other's memory and having memory they're using written over by other programs. With "paging" each program can assume they have all the memory to themselves.

If we were not using virtual memory and my web browser wrote to location 0x200 and my video player then also wrote to 0x200, then the memory would become corrupt. The web browser would have to somehow detect that its memory had been overwritten and fix the issue. This would be a mess and make programming a lot harder, less secure and performant. So, the OS helps us out by doing coordination for us. What the web browser sees as 0x200 may actually be mapped to the physical address 0x100. While what the video player sees as 0x200 may be mapped to the physical address 0x300. Both programs can assume they're the only programming running on a computer with a huge amount of memory, and the OS provides this nice allusion for them.

Make more sense?

from book.

That makes sense yea, it's a lot clearer now. Perhaps the page could be updated with a short explanation about this.

from book.

@LaylConway I can do that. Any insight into which parts of the explanations above really helped this click for you would be awesome.

from book.

So one issue is that paging is technically distinct from virtual memory; it's the way VM is implemented, but they're not the same thing. So I was also trying to be true to that here...

On May 22, 2016, 08:57 -0400, [email protected], wrote:

That makes sense yea, it's a lot clearer now. Perhaps the page could be updated with a short explanation about this.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly orview it on GitHub(#100 (comment))

from book.

Specifically the part that the OS can pretend multiple programs all are the only one running on the machine helped the most with understanding why this is important.

from book.

@steveklabnik that's true, but could we not explain all the reasons for Virtual Memory and then explain that we are using paging to accomplish this?

from book.

We don't even have a user space yet, so we don't have a reason to have virtual memory. So I'm hesitant to explain something at this point when it's a long time off.

That said, this tension is exactly something I struggled with; maybe we shouldn't talk so much about why paging and just present it as something long mode needs?

On May 22, 2016, 09:55 -0400, Ryan [email protected], wrote:

@steveklabnik(https://github.com/steveklabnik)that's true, but could we not explain all the reasons for Virtual Memory and then explain that we are using paging to accomplish this?

—
You are receiving this because you were mentioned.
Reply to this email directly orview it on GitHub(#100 (comment))

from book.

Yeah, I like it. Or "we won't see the full benefit of this until later", something along these lines.

I do agree that hiding too much is very unsatisfying, and kind of the opposite of what the project is about! It's a tough balance.

On May 22, 2016, 12:54 -0400, Ryan [email protected], wrote:

Yea that could work. I find "we just need to do this; don't ask too many questions" very unsatisfying when I read. I like the parts of the book so far that really stress things we need to do because of legacy reasons, but I still feel like it would be better if we had some appendices going into more depth as to why.

For this section, I think it might be appropriate to say something to the effect of "in order to go into full 64-bit mode, we need to set up something called paging. Paging is certain way to implement something called "virtual memory" which will allow the OS to hide the details of the memory layout of the physical machine from programs that run on the computer. The reasons why this is a good idea will come up later in our chapter on 'user spaces'. For now, we'll just take this as something seemingly random we have to do to get into full 64-bit 'long mode'".

Does something like that work? It lets the reader know that this isn't totally random, but that we're not quite ready to fully understand why we need to do this.

—
You are receiving this because you were mentioned.
Reply to this email directly orview it on GitHub(#100 (comment))

from book.

I'm not working on the first edition anymore, so I'm going to give this a close. This stuff isn't in the second edition yet, and it's not likely to make it exactly. If someone wants to send in a PR to fix up the first edition, please feel free!

from book.