niemandhier 5 hours ago

People here keep claiming “Anything is possible with unlimited budget”.

Cerns budget is 1.4 billion Euro, 50 million Euro for all IT infrastructure.

https://cds.cern.ch/record/2888205/files/English.pdf#page18

It’s not the money, it’s the people. Update: Added source.

  • atoav 4 hours ago

    That kind of place can draw a certain kind of employee. This finding is hard to transfer to commercial projects. Sure employees will always claim to be really motivated, especially in the marketing material, but are they we-are-nerds-working-on-the-bleeding-edge-of-human-knowledge-motivated?

    Probably not, but there is surely some manager out there who made themselves believe they can motivate their employees to show the same devotion for the self-made hardships of some mostely pointless SaaS product. If you want to grab that kind of spirit, what you do needs to fundamentally make sense beyond just making somebody money.

    • sligor 2 hours ago

      That's exactly how we were able to go to the moon in 55 years ago. And why it's complicated today. It was of course lot of money. But it was mostly a lot of highly skilled, motivated devoted people doing for an ultimate common goal. Money would not have been sufficient by itself.

    • jedrek 2 hours ago

      Also, CERN does not have a profit motive.

      How much good work have the people reading this thread had to trash because it didn't align with Q3 OKRs? How much time and energy did they put into garbage solutions because they had to hit a KPI by the last day of June?

      • bayindirh 7 minutes ago

        > Also, CERN does not have a profit motive.

        This is a great point. We work with CERN on a project, and we're all volunteers, but we work on something we need, and contribute back to it wholeheartedly.

        At the end of the day, CERN wants to give away project governance to someone else in the group, because they don't want to be the BDFL of anything they help creating. It allows them to pursue new and shiny things and build them.

        There's an air consisting of "it's done when it's done", and "we need this, so we should build this without watering it down", so projects move at a steady pace, but the code and product is always as high quality as possible.

    • quailfarmer 3 hours ago

      That’s a great observation, and I think generally correct, but there are private companies where that sort of motivation exists, for basically the same reason

    • niemandhier an hour ago

      CERN buddy of mine suggested that exposing a colony of physicists to elevated ambient levels of helium would trigger excessive infrastructure building behavior.

  • dauertewigkeit an hour ago

    Good hiring managers can find the hidden gems. These are typically people who don't have the resume to join FAANG immediately, due to lacking the pedigree, but who have lots of potential. Also these same people typically don't last long because they do eventually move on.

    Also it helps that Europe is so behind in tech that if you want to do some cutting edge tech you are almost forced to join a public institution because private ones are not doing anything exciting.

  • lokimedes 5 hours ago

    Also, the in-kind contributions from hundreds of institutes around the world. Much can, and has, been said about physicist code, but CERN is the center of a massive community of “pre-dropout” geniuses. I can’t count the number of former students that later joined Google and the likes. Many are frequenting HN.

  • adev_ 4 hours ago

    CERN was a good example of how much can be done with how little when you have the right people.

    For a long time, the entire Linux distribution (Scientific Linux) used for ~15K collaborators, the infra and the grid computing was managed by a team of around 4-5 people.

    The teams managing the network access (LanDB), the distributed computing system, the scientific framework (ROOT) and the storage are also small, dedicated skilled teams.

    And the result speaks for itself.

    Unfortunately, most of that went to shit quite recently when they replaced the previous head of IT by a Microsoft fanboy/girl coming from outside of the scientific environment. The first thing he/she did was to force Microsoft bloatware everywhere to replace existing working OSS solutions.

    • wuming2 2 hours ago

      > Unfortunately, most of that went to shit quite recently when they replaced the previous head of IT by a Microsoft fanboy(girl?) coming from outside of the scientific environment.

      Painful to read so I did a short check. From a news post I don’t want to link here, but easily found searching “CERN, the famous scientific lab where the web was born, tells us why it's ditching Microsoft and helping others do the same”, direction taken in 2019 seemed quite the opposite. I am not sure how current head of IT at CERN, Enrica Porcari, fits in to the story. Insider info will be appreciated.

      • adev_ 2 hours ago

        > direction taken in 2019 seemed quite the opposite

        The head of IT changed in 2021 if it answers your question.

        • dguest an hour ago

          There was a huge initiative at CERN to move to non-MS products.

          It was great actually: suddenly we were leaving behind a bunch of bloated MS cruft and working with nice stuff. As someone working at CERN I was really inspired, not just by the support for open source but by how well it all worked.

          Then next thing I knew we were doubling down on MS stuff. I don't know what happened. It was sad though, and the user experience did not improve in the end.

          I'm not close enough to CERN-IT to know the details. But for what it's worth, no one I knew in IT could think of a good reason for going back.

        • wuming2 2 hours ago

          Don’t see any previous experience at Microsoft [2]. Just a self taught fan then?

          Edit: “Partnership is the art of understanding shared value. In WFP we have a number of partnerships, not many, but the ones that we have are deep, are sustained, are long-term. And definitely UNICC is one of them. Enrica Porcari, Chief Information Officer and Director Technology Division at the WFP” [1]

          United Nations International Computing Centre (UNICC) is a Microsoft shop. Legit to assume, if OP statement holds true, she got the business sponsorship going while CIO at the World Food Program (WFP).

          This kind of attempted executive takeover is always the strategy of a team. Who sponsored and voted for her at CERN is the real person of interest.

          1. https://www.unicc.org/our-values/what-makes-us-unique/

          2. https://cgnet.com/blog/former-cgnet-employee-enrica-porcari-...

  • amelius 2 hours ago

    > Cerns budget is 1.4 billion Euro

    Kind of weird that a company like Uber has a valuation of $150 billion Euro.

    • dguest 44 minutes ago

      Most of the people who make CERN work aren't working for CERN. The IT department is under CERN, but there are many thousands of "users" who don't get payed by CERN at all. Quite a lot of the fabrication and most of the physics analysis is done by national labs and universities around the world.

      • elashri 39 minutes ago

        CERN budget on experiment level is being paid mostly by contributions from the institutions that is part of this experiment. I am talking about operation, R&D and this would also include personnel contributions to different aspect. There is also service work that each one of the users must do beside doing physics. I am for example work on software development stack beside my current physics analysis. Some of my colleagues working on hardware.

        Then there are country level contributions that pays for CERN infrastructure and maintenance (and inter experiment stuff) and direct employees salaries.

    • gwervc an hour ago

      How many people ordering a meal (often out of laziness) per day vs thinking and searching the mysteries of universe? Economically it makes sense that Uber generates a lot more of cash.

    • yccs27 2 hours ago

      Apples to oranges. Budget is per year, valuation is total.

      A better comparison would be Uber's revenue of $37 billion in 2023.

      • amelius an hour ago

        I don't see why it's Apples to oranges. Uber could pay for 150 CERN-years.

        • chmod775 an hour ago

          No, they could not.

          Valuation is not money in the bank. It does not even represent an amount that is convertible to an equal amount of liquid currency.

          It's a number that is hardly useful for anything and I'm tired of people cooking up all sorts of nonsense with it.

          • amelius 28 minutes ago

            Ok, maybe it's 75 CERN years or maybe even 10. The point still stands.

            PS: Sorry if you got tired, but I'm tired of people explaining what valuation isn't when we're just talking orders of magnitude.

  • rob_c 4 hours ago

    Yes, but that still covers infrastructure (cables) and a lot of equipment for the experiments including but not limited to massive storage and tape backup, distributed local compute, and local cluster management all with users busy trying to pummel it with the latest and greatest ideas of how they can use it faster and better... Not to mention specialist software and licences. 50M doesn't go that far when you factor all of this in

udev4096 5 hours ago

This is fascinating. How are they managing or even taking backup for this gigantic storage?

  • ephimetheus 5 hours ago

    For experiment data, there is a layer on top of all of this that distributes datasets across the computing grid. That system has a way to handle replicate at the dataset level.

  • rob_c 4 hours ago

    Tape and off-site replicas at globally distributed data centres for science. Of the 1EB a huge amount of that is probably in automated recall and replication with "users" running staged processing of the data at different sites ultimately with data being reduced to "manageable" GB-TB level for scientists to do science

    • fnands an hour ago

      Yup, lots of tape for stuff in cold storage, and then some subset of that on disk spread out over several sites.

      It's kinda interesting to watch anything by Alberto Pace, the head of storage at CERN to get an understanding of the challenges and constraints: https://www.youtube.com/watch?v=ym2am-FumXQ

      I was basically on the helpdesk for the system for a few years so had to spend a fair amount of time helping people replicate data from one place to another, or from tape onto disk.

a-dub 6 hours ago

does modern fuse still context switch too much or does it now use io_uring or similar?

  • mappu 5 hours ago

    FUSE over io_uring is still WIP: https://lwn.net/Articles/988186/

    FUSE Passthrough landed in kernel 6.9, which also reduces context switching in some cases: https://www.phoronix.com/news/Linux-6.9-FUSE-Passthrough . The benchmarks in this article are pretty damning for regular FUSE.

    • Dwedit 2 hours ago

      FUSE Passthrough is only useful for filesystems that wrap an existing filesystem, such as union mounts. Otherwise, you don't have an open file to hand over.

    • a-dub 4 hours ago

      yeah but still not great for metadata operations, no?

      i remember it was really not great for large sets of search paths because it defeated the kernel's built-in metadata caches with excessive context switching?

  • Dwedit 6 hours ago

    Last I read about FUSE, adding a 128KB read-ahead buffer drastically reduced context switching.

qwertox 6 hours ago

IIRC I had issues with inotify when I was editing files on a remote machine via SSHFS, when these files were being used inside a Docker container. inotify inside the container did not trigger the notifications, whereas it did, when editing a file with an editor directly on that host.

I think this was related to FUSE, that Docker just didn't get notified.

synicalx 7 hours ago

1EB with only 30k users, thats a wild TB-per-user ratio. My frame of reference; the largest storage platform I've ever worked on was a combined ~60PB (give or take) and that had hundreds of millions of users.

  • chipdart 5 hours ago

    Most humans don't handle sensor and simulation data for a living, though. CERN just so happens to employ thousands who do that for a living.

  • shric 5 hours ago

    My frame of reference; the largest storage platform I've ever worked on was a combined ~tens of EB (give or take) and that had over a billion users.

  • hackernewds 6 hours ago

    That's the scale of the universe, compared to data generated by humans

maybeben 9 hours ago

i mean, they also have one of the largest ceph deployments. anything is scalable with no budget.

  • pas 8 hours ago

    slide 22 states that the cost is 1 CHF/TB/month (on 10+2 erasure coded disks), though it would be interesting to do a breakdown of costs (development, hardware, maintenance, datacenter, servicing, management, etc..)

    • pclmulqdq 6 hours ago

      1 CHF/TB/month is a bit expensive for storage at that scale, so it would definitely be interesting to see what they're spending the money on and what they are (and aren't) counting in that price.

      • rob_c 4 hours ago

        Tape backup, accessibility, networking, availability... At 1CHF/TB that's a lot better than my local university still charging >100x that for such services internally

      • hackernewds 6 hours ago

        No budget often tags along with no accountability

  • hi-v-rocknroll 7 hours ago

    They probably consume Panasas, IBM, DDN, and BeeGFS gear and licensing too.

    • adev_ 4 hours ago

      Nop.

      Most internal data is spread between Ceph and home-made distributed storage system named EOS (https://indico.cern.ch/event/138478/contributions/149912/att...) running over commodity hardware.

      The only commerical-backed storage system is the long term storage tape system. Still it has an home-made overlay API over it to interface with the rest of the systems.

    • rob_c 4 hours ago

      Good god no. Nowhere near anything so crass. CEPH and EOS all the way

vfclists 8 hours ago

[flagged]

  • coherentpony 8 hours ago

    https://en.wikipedia.org/wiki/CERN#Scientific_achievements

    Here's a couple, in case you don't want to read the page:

    - CERN pioneered the introduction of TCP/IP for its intranet, beginning in 1984

    - CERN has developed a number of policies and official documents that enable and promote open science

    - The CERN Science Gateway, opened in October 2023,[179] is CERN's latest facility for science outreach and education

    I purposefully picked items that weren't directly particle physics related.

    • scottapotamas 6 hours ago

      Just tacking some detail onto "promote open science".

      CERN was/is a large early user and supporter of the open source KiCAD electronics CAD tooling. The downstream impact of improved accessibility to solid ECAD tooling has been a large contributing factor to the growing ecosystem of open electronics.

      A lot of really impressive test and measurement equipment to support their research is developed in the open (see https://ohwr.org/project). People on HN are probably most likely to have heard of the White Rabbit timing project, but there's fantastic voltmeter designs, a lot of FPGA projects for carriers, gateware, fun RF designs.

      • snugglebert 4 hours ago

        They also use the expensive big ECAD tools for the super complex stuff.

        But no secret - they are one of the reasons why Kicad isn't an ugly duckling anymore.

  • tiffanyh 8 hours ago

    Inventing WWW is arguably the single greatest economic development in the history of mankind.

    • hollerith 8 hours ago

      But if Berners-Lee hadn't started the WWW, someone else probably would have within a few years: the hard part was the development of the internet, i.e., a flexible low-cost wide-area network where anyone could start a new service (look in /etc/services to see all the services that people have defined over the years) without the need to get permission from anyone.

      IIRC the first WWW server went live in 1990. By then there was already WAIS, Archie and Veronica (search engines for anonymous-FTP sites). In 1991, the first Gopher server went live. Gopher grew rapidly till the late 1990s.

      The US government's Advanced Research Projects Agency started funding research into "packet-switched networks" in 1960 which would eventually lead to the internet, which went live in 1969 (under the name ARPAnet, but only a pedant would say that ARPAnet is not the early verion of the internet). Then the USG continued to fund the internet every year till it no longer needed funding in the early 1990s.

      So, CERN and Berners-Lee (mostly the latter because no one at CERN other than Berners-Lee cared much about the WWW in its early days before it became a big hit) get some credit for the WWW, but in my reckoning it is a small amount of credit.

      • bozhark 7 hours ago

        But if…

        But wasn’t.

  • Enginerrrd 8 hours ago

    A lot of the benefit has come from learning expertise in applications.

    Tons of the data science tools have roots in CERN. Tons of interesting statistical methods, tons of experience R&D with superconductors and all manners of sensors.

    Tons of math/ computation techniques / modeling, etc would not be here without for CERN.

    It would be sort of silly to expect that any of their actual discoveries or tests of the SM would have any actual application, but the ancillary benefits are there.

    • hackernewds 6 hours ago

      Which tons? And why would it be silly? If actual new particles or physical phenomena were found the applications would be trillions

      • adgjlsfhk1 5 hours ago

        a particle that requires 30 km particle accelerator to produce isn't going to have that many applications on earth

  • arlort 5 hours ago

    > practical social and economic benefits to humanity as a whole?

    Why does it have to be practical? Scientific discovery is a perfectly valid end in its own even if it only ever means that we understand the universe better

    The fact that almost always scientific discovery turns out to have practical purposes in the long run (centuries, not decades) is an added bonus.

    It's not like it's a huge expense either. If switzerland decided to it could cover the yearly budget of cern, by itself at the cost of a fraction of a percentage of its gdp alone

  • dekhn 7 hours ago

    There's a number of points to unpack here.

    High energy physics research has contributed some technology with social and economic benefits. Some of that has been direct results coming from pure research into fundamental properties of matter and electromagnetic radiation, some are indirect results that came about because when you build an institute like CERN, it spontaneously generates advances in other areas that solve more general problems (this is known as the "collect a bunch of smart people in a single place, with a lot of resources, to solve a unique problem" strategy). But no, most of the research, pure or applied, has not really had direct practical social and economic benefits to humanity as a whole.

    That's entirely missing the point. We, as a society, have decided that we will balance our economic productivity into several different areas- welfare, infrastructure, military, industry, science/research, technology. We believe that investments in areas of research which have no direct benefit still can have positive outcomes- partly through fundamental discoveries, but also enriching us as a species. We also believe these investments will ensure that we have the freedom to be productive in the future.

    A cynic might even say that CERN has played a critical role in keeping people from working on military applications, or working for the enemy.

    If your criticism (it's hard not to read your comment as an implicit criticism) is that we should invest the results of our productivity more directly into areas which maximize social and economic benefits- sure, this is argued about all the time. The SSC was cancelled, at least partly because people failed to see the value in having a world-class HEP facility in the US.

    • hackernewds 6 hours ago

      Does this seem AI generated? Lists zero tangible contributions besides generalities

      Military research has born aeronautics, manufacturing and industrial tech, GPS and The Internet to state a few commonly known

  • rob_c 4 hours ago

    No, but had cynicism? Off the member states the highest cost per used payer is still less than a bag of peanuts each year and most people with throw that at the TV over whatever upsets them without thinking. It's collective science not big pharma which is soaks tax payer money and then sells the discoveries back to you with 1000x markup. And yes CERN has played an important part in the scientific conversation of where we are in the universe and what is looks like. If you don't think that's important I think flat earth cults are working just as hard to derail conversations they don't want to join in good faith...

InDubioProRubio 3 hours ago

The things you can build when everyone is a rockstar :D