17. Inter-process communications: messages and shared memory
Disadvantage of signals:
- One byte
- Totally asynchronous
- Idempotent (this can be cool sometimes, but)
POSIX messages
Base manpage: mq_overview
What we need for messaging:
- Synchronous
- Can store content
- Can be queued
- Can be prioritized
Every message is delivered over certain queue.
An user can create queue by calling mq_open:
1 #include <mqueue.h> 2 #include <sys/stat.h> 3 #include <fcntl.h> 4 #include <stdio.h> 5 6 int main(int argc, char *argv[]) { 7 mqd_t mqd; 8 struct mq_attr attr; 9 10 attr.mq_maxmsg = 10; 11 attr.mq_msgsize = 2048; 12 mqd = mq_open(argv[1], O_RDWR|O_CREAT|O_EXCL, S_IRUSR|S_IWUSR, attr); 13 14 return 0; 15 }
- Queue is for 10 messages 2048 bytes each
- Queue is creating for read/write, if there's no queue with the same name, or else en error is generated
Omitting O_EXCL allows to re-create a queue with the same name, purging all messages, whit is probably not a good idea
Permissions of the object created are 00600 (rw-------)
To send a message program should open a queue write-only and perform mq_send:
- Priority varies from 0 (lowest) to system-depended maximum (at least 31, 32767 in Linux)
Message content is a byte array, it does not have to be zero-terminating string
POISX queue provides prioritization mechanism. Earliest massage from higher priority messages subset is to be delivered first.
To recieve a message, program has to call mq_receive
1 #include <mqueue.h>
2 #include <fcntl.h>
3 #include <stdio.h>
4 #include <stdlib.h>
5
6 int main(int argc, char *argv[]) {
7 mqd_t mqd;
8 unsigned int prio;
9 void *buf;
10 struct mq_attr attr;
11 ssize_t n;
12
13 mqd = mq_open(argv[1], O_RDONLY);
14
15 mq_getattr(mqd, &attr);
16 buf = malloc(attr.mq_msgsize);
17 n = mq_receive(mqd, buf, attr.mq_msgsize, &prio);
18 printf("Read %ld bytes; priority = %u\n", (long) n, prio);
19 free(buf);
20
21 return 0;
22 }
- Knowing nothing about message size, program must retrieve this value from queue attributes to provide an appropriate space in read buffer.
There's no mechanism of message typification, so only size is printed
To remove a queue call mq_unlink(name)
POSIX message API is implemented in librt library, so compile program with -lrt option.
Notifying
Every mq_receive call returns a message if there's one. If queue is empty, mq_receive() can wait for message or return with fail status, depending on O_NONBLOCK flag.
There's alternate method to notify program by signal: a program calls mq_notify to subscribe on certain queue. Every time message is arrived in queue, the program gets a signal described in mq_notify() and can handle message asynchronously.
See an example in mq_receive.
Linux virtual filesystems
A user can create queue by touch-ing arbitrary file in /dev/mqueue/ directory and unlink it just by removing the file object
Note: mqueue is virtual file system. That means it holds no files on no devices, but emulates file object via OS file interface.
System administrator can also manipulate with /proc/sys/fs/mqueue/* files, which are, again, just implementation of Linux «configuring through virtual filesystem» concept.
Using programs above:
1 $ cc -O0 -g snd_mq.c -lrt -o snd_mq
2 $ cc -O0 -g rec_mq.c -lrt -o rec_mq
3 $ cc -O0 -g crt_mq.c -lrt -o crt_mq
4 $ cc -O0 -g unl_mq.c -lrt -o unl_mq
5 $ ./crt_mq /QWE
6 $ ./snd_mq /QWE 5 QkrQ-V
7 $ ./snd_mq /QWE 10 QkrQ-X
8 $ ./snd_mq /QWE 5 QkrQ-V-II
9 $ ./snd_mq /QWE 3 QkrQ-III
10 $ ./snd_mq /QWE 7 QkrQ-VII
11 $ ls -l /dev/mqueue/
12 итого 0
13 -rw-r--r-- 1 tmpuser tmpuser 80 мар 21 17:52 qwe
14 -rw------- 1 tmpuser tmpuser 80 мар 21 18:37 QWE
15 $ cat /dev/mqueue/QWE
16 QSIZE:37 NOTIFY:0 SIGNO:0 NOTIFY_PID:0
17 $ for n in `seq 5`; do ./rec_mq /QWE; done
18 Read 6 bytes; priority = 10
19 Read 8 bytes; priority = 7
20 Read 6 bytes; priority = 5
21 Read 9 bytes; priority = 5
22 Read 8 bytes; priority = 3
23 $ ./unl_mq /QWE
24 $ ls -l /dev/mqueue/
25 итого 0
26
Viewing a queue virtual object hows it's state.
Memory mapping
Kernel has paging mechanism:
- when memory is limited, some memory pages can be swapped out
- so when a program needs one of them:
TLB produces page miss (no physical memory is provided for the virtual address),
- but then loads corresponded page from disk and links to virtual memory page
- so when a program needs one of them:
If paging out a .text section, there is no need to provide a space on swap, because this data is already on disk — e. g. in the binary program file, from which the process was started.
More general process of mapping file to memory is called memory map. The mmap syscall asks kernel to map selected file to the virtual memory address range. After this done, the range can be used as an ordinary array filled with file's contents. The file has not to be read into memory completely, Linux use paging mechanism to represent corresponded file parts.
This is an example of simple cat analog, that mmaps file and than just writes it to stdout:
1 #include <sys/mman.h>
2 #include <sys/stat.h>
3 #include <stdio.h>
4 #include <fcntl.h>
5
6 int main(int argc, char *argv[]) {
7 char *addr;
8 int fd;
9 struct stat sb;
10
11 fd = open(argv[1], O_RDONLY);
12 fstat(fd, &sb);
13 addr = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
14 fwrite(addr, 1, sb.st_size, stdout);
15 return 0;
16 }
PROT_READ means that mmapped pages can only be read by the program
MAP_PRIVATE means the program observe some fixed state of the file
- write to mmapped area does not change the file itself
- program supposes file can not be changed while mmapped in MAP_PRIVATE mode
fstat is used to determine file size (it discovers other file properties as well)
Shared memory
Base manpage: shm_overview
Multiple processes can have some of their virtual memory pages translated to the same physical page. Then they can communicate through this shared area called shared memory.
POSIX shared memory implemented over mmapped file abstraction.
First we need to open named shared memory object (shared memory analog of queue, shmobj for short). Programs can mmap this object, read and write to it.
1 #include <stdio.h>
2 #include <sys/stat.h>
3 #include <fcntl.h>
4 #include <unistd.h>
5 #include <sys/mman.h>
6 #include <stdlib.h>
7
8 int main(int argc, char *argv[]) {
9 int fd;
10 size_t size;
11 void *addr;
12
13 fd = shm_open(argv[1], O_RDWR|O_CREAT|O_EXCL, S_IRUSR|S_IWUSR);
14 size = atol(argv[2]);
15 ftruncate(fd, size);
16
17 addr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
18 close(fd);
19
20 return 0;
21 }
- Modes and permissions of the object are the same as when creating aqueue
There's no sense in having newly created shmobj size other than zero, so ftruncate() call.
We call mmap() for declaring the object shared (with MAP_SHARED, of course)
To write to the shared memory, program opens shmobj, mmaps it and uses the memory as ordinary array
1 #include <stdio.h>
2 #include <fcntl.h>
3 #include <sys/mman.h>
4 #include <string.h>
5 #include <unistd.h>
6
7 int main(int argc, char *argv[]) {
8 int fd;
9 size_t len;
10 char *addr;
11
12 fd = shm_open(argv[1], O_RDWR, 0);
13 len = strlen(argv[2]);
14 ftruncate(fd, len);
15
16 addr = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
17 close(fd);
18
19 printf("Copying %d bytes\n", len);
20 memcpy(addr, argv[2], len);
21 return 0;
22 }
There's no difference if you open shmobj for reading/writing or just writing, it's memory
The smobj descriptor only needed when opening shmobj, we can close it just after mmap()
To read from shared memory, the program opens shmobj and tread it like mmapped file:
1 #include <stdio.h>
2 #include <fcntl.h>
3 #include <sys/mman.h>
4 #include <sys/stat.h>
5 #include <unistd.h>
6
7 int main(int argc, char *argv[]) {
8 int fd;
9 char *addr;
10 struct stat sb;
11
12 fd = shm_open(argv[1], O_RDONLY, 0);
13 fstat(fd, &sb);
14 addr = mmap(NULL, sb.st_size, PROT_READ, MAP_SHARED, fd, 0);
15 close(fd);
16
17 fwrite(addr, 1, sb.st_size, stdout);
18 printf("\n… Done");
19 return 0;
20 }
Note fstat can be used to determine shared memory size as well as to determine file size
As usual, to stop using shmobj, one shall unlink it with shm_unlink(name)
Linux virtual filesystems
Like mqueue, shared memory objects can be created, viewed and unlinked via /dev/shm filesystem. Moreover, viewing smobj is just dumping it's content.
1 $ cc -O0 -g crt_shm.c -lrt -o crt_shm
2 $ cc -O0 -g wrt_shm.c -lrt -o wrt_shm
3 $ cc -O0 -g rd_shm.c -lrt -o rd_shm
4 $ cc -O0 -g unl_shm.c -lrt -o unl_shm
5 $ ./crt_shm /shmem 0
6 Addr: 0xffffffff
7 $ ls -l /dev/shm/
8 итого 0
9 -rw------- 1 tmpuser tmpuser 0 мар 21 20:17 shmem
10 $ ./wrt_shm /shmem 'Qkrq!'
11 Copying 5 bytes
12 $ cat /dev/shm/shmem; echo
13 Qkrq!
14 $ ./rd_shm /shmem
15 Address: 0x77cd0000
16 Qkrq!
17 … Done
18 $ ./unl_shm /shmem
19 $ ls -l /dev/shm/
20 итого 0
21