Lecture 14

High-level Languages and Operating System.

Lecture

OS and Programming Languages

Linux:

• Kernel — C
• Core system libraries — C
• Core system utilities — mostly C and shell
  • Can be C++, Python and Perl
• Scripting and integrating
  • Mostly shell, but Perl was popular and Python is now
  • Sometimes C++
• Almost any OS distribution — ‘‘all’’ kinds of programming languages

⇒ Is C the only system programming language?

Windows:

• Core, kernel and libraries: C++ (seldom C)
• Scripting and integrating:
  • Any .Net-based platform (mostly C#)
  • C++
  • Powershell (was: cmd)
  • JScript, VBScript (Windows Script Host)
• Applications: C++, .Net, almost any 3d-party languages

MacOS:

• Kernel — C
• Core system and libraries — Objective C and C, seldom C++
  • Now: Swift
• Scripting and integration: AppleScript and Shell
• Applications: Swift, Objective C, maybe C++, almost any 3d-party languages

Python and OS Programming

• Cross-platformness
  • +/-
  • ⇒ own implementation of OS features
• POSIX-oriented (=> Linux)
  • +/-
  • ⇒ own implementation of non-POSIX OS features
• High-level, but not OS-oriented (unlike shell)
  • +/-
  • ⇒ own implementation of OS features
• Has a lot of non-privileged syscalls wrappers
• Incredible amount of modules at PyPi
  • Including system-oriented
  • Including service-oriented
• Non «resource scrimp» style
  • If resources do not multiply, use it as a whole, not piece by peice
    • E. g. .read() instead of .readline()
  • no needs to count CPU circles :)
  • …

Modules os ans sys

• Cross-platform path: os.path and pathlib
  • .is*(), .exists() etc.
• os:
  • .environ
  • syscalls wrappers (.fork, .getpid, .fstat, .popen, .wait, almost any! …)
• sys: .executable, .argv, .stdin, .stdout, .stderr, …
• Raising a level example: time → datetime → calendar

Also:

• platform
• …

Subprocess

Concept: cross-platform process execution with communication and exit status control.

• Just run and get a result: run()
  • capture_output/input=; stdin=/stdout=/stderr=
  • …
• High-level popen analog: Popen()
  • example

    from subprocess import *
    # Run first process that outputs to the unnamed pipe
    p1 = Popen(["cal -s"], stdout=PIPE)
    # Run second process that inputs from the other end of the pipe opened
    # and outputs to the second pipe
    p2 = Popen(["hexdump", "-C"], stdin=p1.stdout, stdout=PIPE)
    # Allow p1 to receive a SIGPIPE if p2 exits
    p1.stdout.close()
    # Read from the second pipe (stdout, stderr),
    # but stderr will be empty because no redirection is used
    res = p2.communicate()
    # Note data is bytes, not str
    print(res[0].decode())
    

  • do not use os.system(), it is platform-dependent and unsafe

Multiprocessing

About multithread programming in Python:

• It exists
• It almost non-parallel because of Global_interpreter_lock
  • There is no apparent way to eliminate GIL without significant slowing single-threaded code
• You can use multithread if:
  1. Only one thread eats CPU, but other ones perform I/O
  2. You have complex code design based on threads and significant amount of permanent usage of joint resources
• Unlike C programs, Python ones have no actual difference between threaded and multiprocess design
• Paper about Python GIL

The multiprocessing module:

• Crossplatformness
  • Linux: fork() is used
• Child process is running ‘‘a function’’ (unlike classic fork(), which defines two equal processes)
  • That simplifies data transfer down to the arguments/return in simple cases
• Processes can communicate through special socket-like high-level objects or object queue
• Processes can use high-level shared memory-alike objects or object manager (the last is slower, but can work over network!)
• Processes can be orchestered into a pool running exactly N processs in parallel.
  1. Exact N child processes (called workers) are started
  2. Each worker can execute a function given multiple times as pool flows
     • No other start/stop actions is performed
  3. Workers are stopped when pool is empty

Workshop

Workshop